mlsanityData to 14 Oct 2026

Towards an Understanding of Default Policies in Multitask Policy Optimization

arXiv 2111.02994 ↗First posted 4 Nov 2021cs.LGcited by 8

Not on the map. Why?

Abstract (arXiv)

Much of the recent success of deep reinforcement learning has been driven by regularized policy optimization (RPO) algorithms with strong performance across multiple domains. In this family of methods, agents are trained to maximize cumulative reward while penalizing deviation in behavior from some reference, or default policy. In addition to empirical success, there is a strong theoretical foundation for understanding RPO methods applied to single tasks, with connections to natural gradient, trust region, and variational approaches. However, there is limited formal understanding of desirable properties for default policies in the multitask setting, an increasingly important domain as the field shifts towards training more generally capable agents. Here, we take a first step towards filling this gap by formally linking the quality of the default policy to its effect on optimization. Using these results, we then derive a principled RPO algorithm for multitask learning with strong performance guarantees.

Key facts. Cited by 8 papers in the map (data through 14 Oct 2026). Builds on 30 of its 51 references that are in the map. First posted on arXiv 4 Nov 2021.

Builds on

30 of 51 references.

References · 30

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  3. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  4. 2015Trust Region Policy Optimizationcited by 2,580
  5. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,258
  6. 2017Gradient Episodic Memory for Continual Learningcited by 1,898
  7. 2019Dota 2 with Large Scale Deep Reinforcement Learningcited by 1,103
  8. 2018IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architecturescited by 883
  9. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 782
  10. 2018Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Reviewcited by 482
  11. 2018Maximum a Posteriori Policy Optimisationcited by 328
  12. 2017Distral: Robust Multitask Reinforcement Learningcited by 233
18 more
  1. 2019On the Theory of Policy Gradient Methods: Optimality, Approximation, and Distribution Shiftcited by 177
  2. 2018Understanding the impact of entropy on policy optimizationcited by 176
  3. 2017Equivalence Between Policy Gradients and Soft Q-Learningcited by 173
  4. 2018Learning by Playing - Solving Sparse Reward Tasks from Scratchcited by 167
  5. 2015Taming the Noise in Reinforcement Learning via Soft Updatescited by 127
  6. 2021Contrastive Behavioral Similarity Embeddings for Generalization in Reinforcement Learningcited by 123
  7. 2019V-MPO: On-Policy Maximum a Posteriori Policy Optimization for Discrete and Continuous Controlcited by 85
  8. 2019InfoBot: Transfer and Exploration via the Information Bottleneckcited by 78
  9. 2020Monte-Carlo Tree Search as Regularized Policy Optimizationcited by 61
  10. 2019Information asymmetry in KL-regularized RLcited by 53
  11. 2019Learning to Score Behaviors for Guided Policy Optimizationcited by 34
  12. 2020Making Sense of Reinforcement Learning and Probabilistic Inferencecited by 30
  13. 2020Efficient Wasserstein Natural Gradients for Reinforcement Learningcited by 23
  14. 2020Stable Policy Optimization via Off-Policy Divergence Regularizationcited by 16
  15. 2021Co-Adaptation of Algorithmic and Implementational Innovations in Inference-based Deep Reinforcement Learningcited by 15
  16. 2021A First-Occupancy Representation for Reinforcement Learningcited by 14
  17. 2020The Variational Bandwidth Bottleneck: Stochastic Evaluation on an Information Budgetcited by 13
  18. 2020Iterative Amortized Policy Optimizationcited by 13

Built on by

Cited by year

Citing papers per year: 2022: 5; 2023: 2; 2024: 0; 2025: 1.

8 papers.

  1. 2023Confronting Reward Model Overoptimization with Constrained RLHFcited by 96
  2. 2022Reincarnating Reinforcement Learning: Reusing Prior Computation to Accelerate Progresscited by 65
  3. 2023ReLOAD: Reinforcement Learning with Optimistic Ascent-Descent for Last-Iterate Convergence in Constrained MDPscited by 20
  4. 2022Understanding the Complexity Gains of Single-Task RL with a Curriculumcited by 17
  5. 2022Joint Representation Training in Sequential Tasks with Shared Structurecited by 5
  6. 2025Ctrl-DNA: Controllable Cell-Type-Specific Regulatory DNA Design via Constrained RLcited by 5
  7. 2022Minimum Description Length Controlcited by 0
  8. 2022Transfer RL via the Undo Maps Formalismcited by 0