Data to 7 Oct 2026

Information asymmetry in KL-regularized RL

arXiv 1905.01240 ↗First posted 3 May 2019cs.LG cs.AI stat.MLcited by 53

Not on the map. Why?

Abstract (arXiv)

Many real world tasks exhibit rich structure that is repeated across different parts of the state space or in time. In this work we study the possibility of leveraging such repeated structure to speed up and regularize learning. We start from the KL regularized expected reward objective which introduces an additional component, a default policy. Instead of relying on a fixed default policy, we learn it from data. But crucially, we restrict the amount of information the default policy receives, forcing it to learn reusable behaviors that help the policy learn faster. We formalize this strategy and discuss connections to information bottleneck approaches and to the variational EM algorithm. We present empirical results in both discrete and continuous action domains and demonstrate that, for certain tasks, learning a default policy alongside the policy can significantly speed up and improve learning.

Key facts. Cited by 53 papers in the map (data through 7 Oct 2026). First posted on arXiv 3 May 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2020: 1; 2021: 14; 2022: 20; 2023: 11; 2024: 4; 2025: 1; 2026: 2.

53 papers.

Sort:

By area:

2020Keep Doing What Worked: Behavioral Modelling Priors for Offline Reinforcement Learningcited by 213
2021Goal-Conditioned Reinforcement Learning with Imagined Subgoalscited by 129
2023Learning Agile Soccer Skills for a Bipedal Robot with Deep Reinforcement Learningcited by 111
2023Confronting Reward Model Overoptimization with Constrained RLHFcited by 96
2021From Motor Control to Team Play in Simulated Humanoid Footballcited by 87
2024From Imitation to Refinement -- Residual RL for Precise Assemblycited by 80
2022Retrieval-Augmented Reinforcement Learningcited by 53
2021Creating Multimodal Interactive Agents with Imitation and Self-Supervised Learningcited by 40
2022Imitate and Repurpose: Learning Reusable Robot Movement Skills From Human and Animal Behaviorscited by 27
2021Learning Transferable Motor Skills with Hierarchical Latent Mixture Policiescited by 26
2022Model-Based Offline Reinforcement Learning with Pessimism-Modulated Dynamics Beliefcited by 22
2021An Information-Theoretic Perspective on Credit Assignment in Reinforcement Learningcited by 20

38 more

2022Robot Learning of Mobile Manipulation with Reachability Behavior Priorscited by 20
2021Bayesian Controller Fusion: Leveraging Control Priors in Deep Reinforcement Learning for Roboticscited by 18
2022On Pathologies in KL-Regularized Reinforcement Learning from Expert Demonstrationscited by 18
2021Outcome-Driven Reinforcement Learning via Variational Inferencecited by 17
2021Collect & Infer -- a fresh look at data-efficient Reinforcement Learningcited by 17
2021Evaluating model-based planning and planner amortization for continuous controlcited by 15
2021Unsupervised Domain Adaptation with Dynamics-Aware Rewards in Reinforcement Learningcited by 15
2022Risk-Sensitive Reinforcement Learning with Exponential Criteriacited by 15
2024Learning Robot Soccer from Egocentric Vision with Deep Reinforcement Learningcited by 14
2022Temporal Latent Bottleneck: Synthesis of Fast and Slow Processing Mechanisms in Sequence Learningcited by 13
2023Knowledge Distillation Performs Partial Variance Reductioncited by 11
2023Inverse Decision Modeling: Learning Interpretable Representations of Behaviorcited by 10
2022Hierarchically Structured Task-Agnostic Continual Learningcited by 9
2024Offline Safe Reinforcement Learning Using Trajectory Classificationcited by 9
2021Towards an Understanding of Default Policies in Multitask Policy Optimizationcited by 8
2023Wasserstein Gradient Flows for Optimizing Gaussian Mixture Policiescited by 8
2021Mixture-of-Variational-Experts for Continual Learningcited by 7
2022Complex behavior from intrinsic motivation to occupy action-state path spacecited by 7
2021Toward Robust Long Range Policy Transfercited by 6
2022Reinforcement Learning Your Way: Agent Characterization through Policy Regularizationcited by 6
2022On Rate-Distortion Theory in Capacity-Limited Cognition & Reinforcement Learningcited by 6
2022SkillS: Adaptive Skill Sequencing for Efficient Temporally-Extended Explorationcited by 6
2023Leveraging Jumpy Models for Planning and Fast Learning in Robotic Domainscited by 6
2021On Multi-objective Policy Optimization as a Tool for Reinforcement Learning: Case Studies in Offline RL and Finetuningcited by 4
2022Data augmentation for efficient learning from parametric expertscited by 4
2023Bayesian Reinforcement Learning with Limited Cognitive Loadcited by 4
2024Satisficing Exploration for Deep Reinforcement Learningcited by 4
2022Priors, Hierarchy, and Information Asymmetry for Skill Transfer in Reinforcement Learningcited by 3
2022SFP: State-free Priors for Exploration in Off-Policy Reinforcement Learningcited by 3
2022Towards Artificial Virtuous Agents: Games, Dilemmas and Machine Learningcited by 3
2025Mutual Information Tracks Policy Coherence in Reinforcement Learningcited by 3
2026IMWM: Intuition Models Complement World Models for Latent Planningcited by 3
2022Reinforcement Learning with Intrinsic Affinity for Personalized Prosperity Managementcited by 1
2023A general Markov decision process formalism for action-state entropy-regularized reward maximizationcited by 1
2023Guaranteed Trust Region Optimization via Two-Phase KL Penalizationcited by 1
2022Coordinating Policies Among Multiple Agents via an Intelligent Communication Channelcited by 0
2022Minimum Description Length Controlcited by 0
2022Symbolic Explanation of Affinity-Based Reinforcement Learning Agents with Markov Modelscited by 0

50 most cited of 53 All 53 (JSON)