Marginalized operators for off-policy reinforcement learning
2022 · in (AISTATS 2022) and (ICML 2021 - RL Theory)
Abstract
In this work, we propose marginalized operators, a new class of off-policy evaluation operators for reinforcement learning. Marginalized operators strictly generalize generic multi-step operators, such as Retrace, as special cases. Marginalized operators also suggest a form of sample-based estimates with potential variance reduction, compared to sample-based estimates of the original multi-step operators. We show that the estimates for marginalized operators can be computed in a scalable way, which also generalizes prior results on marginalized importance sampling as special cases. Finally, we empirically demonstrate that marginalized operators provide performance gains to off-policy evaluation and downstream policy optimization algorithms.
PDF · International Conference on Artificial Intelligence and Statistics · arXiv preprint · bibtex · DOI