mlsanityData to 7 Oct 2026

A State-Distribution Matching Approach to Non-Episodic Reinforcement Learning

arXiv 2205.05212 ↗First posted 11 May 2022cs.LG cs.AI cs.ROcited by 21

Not on the map. Why?

Abstract (arXiv)

While reinforcement learning (RL) provides a framework for learning through trial and error, translating RL algorithms into the real world has remained challenging. A major hurdle to real-world application arises from the development of algorithms in an episodic setting where the environment is reset after every trial, in contrast with the continual and non-episodic nature of the real-world encountered by embodied agents such as humans and robots. Prior works have considered an alternating approach where a forward policy learns to solve the task and the backward policy learns to reset the environment, but what initial state distribution should the backward policy reset the agent to? Assuming access to a few demonstrations, we propose a new method, MEDAL, that trains the backward policy to match the state distribution in the provided demonstrations. This keeps the agent close to the task-relevant states, allowing for a mix of easy and difficult starting states for the forward policy. Our experiments show that MEDAL matches or outperforms prior methods on three sparse-reward continuous control tasks from the EARL benchmark, with 40% gains on the hardest task, while making fewer assumptions than prior works.

Key facts. Cited by 21 papers in the map (data through 7 Oct 2026). Builds on 34 of its 51 references that are in the map. First posted on arXiv 11 May 2022.

Builds on

34 of 51 references.

References · 34

  1. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,155
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  3. 2016Generative Adversarial Imitation Learningcited by 1,576
  4. 2017Hindsight Experience Replaycited by 1,096
  5. 2018Exploration by Random Network Distillationcited by 966
  6. 2018Diversity is All You Need: Learning Skills without a Reward Functioncited by 734
  7. 2017Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrationscited by 685
  8. 2016f-GAN: Training Generative Neural Samplers using Variational Divergence Minimizationcited by 504
  9. 2016Guided Cost Learning: Deep Inverse Optimal Control via Policy Optimizationcited by 400
  10. 2017Overcoming Exploration in Reinforcement Learning with Demonstrationscited by 322
  11. 2019Dynamics-Aware Unsupervised Discovery of Skillscited by 308
  12. 2017Leveraging Demonstrations for Deep Reinforcement Learning on Robotics Problems with Sparse Rewardscited by 302
22 more
  1. 2016NIPS 2016 Tutorial: Generative Adversarial Networkscited by 284
  2. 2016Variational Intrinsic Controlcited by 262
  3. 2018Provably Efficient Maximum Entropy Explorationcited by 240
  4. 2018Discriminator-Actor-Critic: Addressing Sample Inefficiency and Reward Bias in Adversarial Imitation Learningcited by 203
  5. 2019A Divergence Minimization Perspective on Imitation Learning Methodscited by 188
  6. 2019Efficient Exploration via State Marginal Matchingcited by 186
  7. 2020Explore, Discover and Learn: Unsupervised Discovery of State-Covering Skillscited by 120
  8. 2019End-to-End Robotic Reinforcement Learning without Reward Engineeringcited by 115
  9. 2020The Ingredients of Real-World Robotic Reinforcement Learningcited by 111
  10. 2020Learning to Walk in the Real World with Minimal Human Effortcited by 103
  11. 2021Reset-Free Reinforcement Learning via Multi-Task Learning: Learning Dexterous Manipulation Behaviors without Human Interventioncited by 90
  12. 2017Leave no Trace: Learning to Reset for Safe and Autonomous Reinforcement Learningcited by 86
  13. 2021Off-Policy Imitation Learning from Observationscited by 70
  14. 2018Variational Inverse Control with Events: A General Framework for Data-Driven Reward Definitioncited by 66
  15. 2021Legged Robots that Keep on Learning: Fine-Tuning Locomotion Policies in the Real Worldcited by 64
  16. 2021Visual Adversarial Imitation Learning using Variational Modelscited by 42
  17. 2020Reset-Free Lifelong Learning with Skill-Space Planningcited by 38
  18. 2021Autonomous Reinforcement Learning: Formalism and Benchmarkingcited by 33
  19. 2021Autonomous Reinforcement Learning via Subgoal Curriculacited by 32
  20. 2020Continual Learning of Control Primitives: Skill Discovery via Reset-Gamescited by 29
  21. 2016Reset-free Trial-and-Error Learning for Robot Damage Recoverycited by 24
  22. 2020Ecological Reinforcement Learningcited by 13

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 3; 2023: 8; 2024: 6; 2025: 1; 2026: 3.

21 papers.

Sort:

By area:

  1. 2023Robot Fine-Tuning Made Easy: Pre-Training Rewards and Policies for Autonomous Real-World Reinforcement Learningcited by 49
  2. 2023Self-Improving Robots: End-to-End Autonomous Visuomotor Reinforcement Learningcited by 29
  3. 2022You Only Live Once: Single-Life Reinforcement Learningcited by 23
  4. 2022When to Ask for Help: Proactive Interventions in Autonomous Reinforcement Learningcited by 20
  5. 2023Autonomous Robotic Reinforcement Learning with Asynchronous Human Feedbackcited by 12
  6. 2024Hybrid Reinforcement Learning from Offline Observation Alonecited by 5
  7. 2024Intelligent Switching for Reset-Free RLcited by 4
  8. 2024Improving Offline Reinforcement Learning with Inaccurate Simulatorscited by 4
  9. 2026UniIntervene: Agentic Intervention for Efficient Real-World Reinforcement Learningcited by 4
  10. 2023Provable Reset-free Reinforcement Learning by No-Regret Reductioncited by 3
  11. 2023When Learning Is Out of Reach, Reset: Generalization in Autonomous Visuomotor Reinforcement Learningcited by 3
  12. 2024NeoRL: Efficient Exploration for Nonepisodic RLcited by 3
9 more
  1. 2024Reset-free Reinforcement Learning with World Modelscited by 3
  2. 2023Demonstration-free Autonomous Reinforcement Learning via Implicit and Bidirectional Curriculumcited by 2
  3. 2023SeRO: Self-Supervised Reinforcement Learning for Recovery from Out-of-Distribution Situationscited by 2
  4. 2025Auto-exploration for online reinforcement learningcited by 2
  5. 2026What Matters for Simulation to Online Reinforcement Learning on Real Robotscited by 2
  6. 2022Age of Semantics in Cooperative Communications: To Expedite Simulation Towards Real via Offline Reinforcement Learningcited by 0
  7. 2023Learning to Stabilize Online Reinforcement Learning in Unbounded State Spacescited by 0
  8. 2024Preference-Guided Reinforcement Learning for Efficient Explorationcited by 0
  9. 2026REVERSAL-BENCH: A Reversibility Axis and Reset Oracle for Measuring the Reset-Free RL Cliffcited by 0