mlsanityData to 14 Oct 2026

Safe Exploration by Solving Early Terminated MDP

arXiv 2107.04200 ↗First posted 9 Jul 2021cs.LGcited by 15

Not on the map. Why?

Abstract (arXiv)

Safe exploration is crucial for the real-world application of reinforcement learning (RL). Previous works consider the safe exploration problem as Constrained Markov Decision Process (CMDP), where the policies are being optimized under constraints. However, when encountering any potential dangers, human tends to stop immediately and rarely learns to behave safely in danger. Motivated by human learning, we introduce a new approach to address safe RL problems under the framework of Early Terminated MDP (ET-MDP). We first define the ET-MDP as an unconstrained MDP with the same optimal value function as its corresponding CMDP. An off-policy algorithm based on context models is then proposed to solve the ET-MDP, which thereby solves the corresponding CMDP with better asymptotic performance and improved learning efficiency. Experiments on various CMDP tasks show a substantial improvement over previous methods that directly solve CMDP.

Key facts. Cited by 15 papers in the map (data through 14 Oct 2026). Builds on 34 of its 44 references that are in the map. First posted on arXiv 9 Jul 2021.

Builds on

34 of 44 references.

References · 34

  1. 2017Attention Is All You Needcited by 60,361
  2. 2017Proximal Policy Optimization Algorithmscited by 15,215
  3. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,897
  4. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  5. 2014Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translationcited by 4,057
  6. 2015Trust Region Policy Optimizationcited by 2,580
  7. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,417
  8. 2016Concrete Problems in AI Safetycited by 1,642
  9. 2018On First-Order Meta-Learning Algorithmscited by 1,018
  10. 2017Constrained Policy Optimizationcited by 983
  11. 2017Evolution Strategies as a Scalable Alternative to Reinforcement Learningcited by 574
  12. 2019Efficient Off-Policy Meta-Reinforcement Learning via Probabilistic Context Variablescited by 389
22 more
  1. 2016Benchmarking Deep Reinforcement Learning for Continuous Controlcited by 371
  2. 2015Risk-Constrained Reinforcement Learning with Percentile Risk Criteriacited by 337
  3. 2020Responsive Safety in Reinforcement Learning by PID Lagrangian Methodscited by 316
  4. 2018A Lyapunov-based Approach to Safe Reinforcement Learningcited by 297
  5. 2019End-to-End Safe Reinforcement Learning through Barrier Functions for Safety-Critical Continuous Control Taskscited by 289
  6. 2018Learning to Drive in a Daycited by 273
  7. 2018Safe Exploration in Continuous Action Spacescited by 247
  8. 2019Benchmarking Model-Based Reinforcement Learningcited by 162
  9. 2019IPO: Interior-point Policy Optimization under Constraintscited by 132
  10. 2020Learning to be Safe: Deep RL with a Safety Criticcited by 116
  11. 2020Safe Reinforcement Learning in Constrained Markov Decision Processescited by 115
  12. 2020Conservative Safety Critics for Explorationcited by 113
  13. 2017Trial without Error: Towards Safe Reinforcement Learning via Human Interventioncited by 103
  14. 2017Time Limits in Reinforcement Learningcited by 79
  15. 2019Learning for Safety-Critical Control with Control Barrier Functionscited by 69
  16. 2019Meta-Q-Learningcited by 62
  17. 2020Cautious Adaptation For Reinforcement Learning in Safety-Critical Settingscited by 42
  18. 2019Open-Sourced Reinforcement Learning Environments for Surgical Roboticscited by 26
  19. 2021Lyapunov Barrier Policy Optimizationcited by 12
  20. 2018Constrained Exploration and Recovery from Experience Shapingcited by 3
  21. 2020Online Constrained Model-based Reinforcement Learningcited by 3
  22. 2019Visualizing Movement Control Optimization Landscapescited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 1; 2022: 1; 2023: 4; 2024: 3; 2025: 2; 2026: 4.

15 papers.

Sort:

By area:

  1. 2021Recurrent Model-Free RL Can Be a Strong Baseline for Many POMDPscited by 122
  2. 2022RORL: Robust Offline Reinforcement Learning via Conservative Smoothingcited by 84
  3. 2023OmniSafe: An Infrastructure for Accelerating Safe Reinforcement Learning Researchcited by 80
  4. 2023Safe Exploration in Reinforcement Learning: A Generalized Formulation and Algorithmscited by 28
  5. 2024MetaUrban: An Embodied AI Simulation Platform for Urban Micromobilitycited by 21
  6. 2023Reinforcement Learning in the Era of LLMs: What is Essential? What is needed? An RL Perspective on RLHF, Prompting, and Beyondcited by 20
  7. 2024A Review of Safe Reinforcement Learning Methods for Modern Power Systemscited by 11
  8. 2023Accountability in Offline Reinforcement Learning: Explaining Decisions with a Corpus of Examplescited by 9
  9. 2025The Courage to Stop: Overcoming Sunk Cost Fallacy in Deep Reinforcement Learningcited by 6
  10. 2026Safe Exploration via Policy Priorscited by 6
  11. 2024Safe Reinforcement Learning for Power System Control: A Reviewcited by 3
  12. 2025UrbanVLA: A Vision-Language-Action Model for Urban Micromobilitycited by 3
3 more
  1. 2026Off-Policy Safe Reinforcement Learning with Constrained Optimistic Explorationcited by 2
  2. 2026Utility-Constrained Policy Optimizationcited by 0
  3. 2026Turning Safety into Competence: Minimally Exploitable Robot Policies via Safety-Filtered Reinforcement Learningcited by 0