mlsanityData to 7 Oct 2026

A Comprehensive Survey on Inverse Constrained Reinforcement Learning: Definitions, Progress and Challenges

arXiv 2409.07569 ↗First posted 11 Sep 2024cs.LG cs.AIcited by 10

Not on the map. Why?

Abstract (arXiv)

Inverse Constrained Reinforcement Learning (ICRL) is the task of inferring the implicit constraints that expert agents adhere to, based on their demonstration data. As an emerging research topic, ICRL has received considerable attention in recent years. This article presents a categorical survey of the latest advances in ICRL. It serves as a comprehensive reference for machine learning researchers and practitioners, as well as starters seeking to comprehend the definitions, advancements, and important challenges in ICRL. We begin by formally defining the problem and outlining the algorithmic framework that facilitates constraint inference across various scenarios. These include deterministic or stochastic environments, environments with limited demonstrations, and multiple agents. For each context, we illustrate the critical challenges and introduce a series of fundamental methods to tackle these issues. This survey encompasses discrete, virtual, and realistic environments for evaluating ICRL agents. We also delve into the most pertinent applications of ICRL, such as autonomous driving, robot control, and sports analytics. To stimulate continuing research, we conclude the survey with a discussion of key unresolved questions in ICRL that can effectively foster a bridge between theoretical understanding and practical industrial applications. The papers referenced in this survey can be found at https://github.com/Jasonxu1225/Awesome-Constraint-Inference-in-RL.

Key facts. Cited by 10 papers in the map (data through 7 Oct 2026). Builds on 69 of its 121 references that are in the map. First posted on arXiv 11 Sep 2024.

Builds on

69 of 121 references.

References · 69

  1. 2017Attention Is All You Needcited by 59,932
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  4. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,873
  5. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,777
  6. 2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,565
  7. 2017Masked Autoregressive Flow for Density Estimationcited by 680
  8. 2017Reinforcement Learning with Deep Energy-Based Policiescited by 659
  9. 2023Safe RLHF: Safe Reinforcement Learning from Human Feedbackcited by 630
  10. 2019Multi-Agent Reinforcement Learning: A Selective Overview of Theories and Algorithmscited by 555
  11. 2017Learning Robust Rewards with Adversarial Inverse Reinforcement Learningcited by 485
  12. 2024RoboCasa: Large-Scale Simulation of Everyday Tasks for Generalist Robotscited by 391
38 more
  1. 2020Is Pessimism Provably Efficient for Offline RL?cited by 362
  2. 2021Safe Learning in Robotics: From Learning-Based Control to Safe Reinforcement Learningcited by 337
  3. 2019Reinforcement Learning in Healthcare: A Surveycited by 308
  4. 2021GFlowNet Foundationscited by 286
  5. 2018The highD Dataset: A Drone Dataset of Naturalistic Vehicle Trajectories on German Highways for Validation of Highly Automated Driving Systemscited by 276
  6. 2021Bellman-consistent Pessimism for Offline Reinforcement Learningcited by 267
  7. 2023Certifying LLM Safety against Adversarial Promptingcited by 248
  8. 2019INTERACTION Dataset: An INTERnational, Adversarial and Cooperative moTION Dataset in Interactive Driving Scenarios with Semantic Mapscited by 231
  9. 2018Learning from positive and unlabeled data: a surveycited by 218
  10. 2023RoboGen: Towards Unleashing Infinite Data for Automated Robot Learning via Generative Simulationcited by 206
  11. 2021IQ-Learn: Inverse soft-Q Learning for Imitationcited by 191
  12. 2021DC3: A learning method for optimization with hard constraintscited by 181
  13. 2018Understanding Measures of Uncertainty for Adversarial Example Detectioncited by 177
  14. 2019Constrained Reinforcement Learning Has Zero Duality Gapcited by 144
  15. 2019The inD Dataset: A Drone Dataset of Naturalistic Road User Trajectories at German Intersectionscited by 137
  16. 2021OptiDICE: Offline Policy Optimization via Stationary Distribution Correction Estimationcited by 121
  17. 2020Near-Optimal Reinforcement Learning with Self-Playcited by 108
  18. 2022A Survey on Safety-Critical Driving Scenario Generation -- A Methodological Perspectivecited by 100
  19. 2021Online Robust Reinforcement Learning with Model Uncertaintycited by 95
  20. 2022Robust Reinforcement Learning using Offline Datacited by 95
  21. 2018Actions Speak Louder Than Goals: Valuing Player Actions in Soccercited by 90
  22. 2023Diffusion Models for Reinforcement Learning: A Surveycited by 84
  23. 2022Supported Policy Optimization for Offline Reinforcement Learningcited by 73
  24. 2021When Can We Learn General-Sum Markov Games with a Large Number of Players Sample-Efficiently?cited by 63
  25. 2023Dual RL: Unification and New Methods for Reinforcement and Imitation Learningcited by 48
  26. 2024Enhancing LLM Safety via Constrained Direct Preference Optimizationcited by 47
  27. 2020Inverse Constrained Reinforcement Learningcited by 45
  28. 2022Efficient Learning of Safe Driving Policy via Human-AI Copilot Optimizationcited by 44
  29. 2019Dirichlet Variational Autoencodercited by 43
  30. 2019Maximum Likelihood Constraint Inference for Inverse Reinforcement Learningcited by 33
  31. 2020Robust Inverse Reinforcement Learning under Transition Dynamics Mismatchcited by 33
  32. 2022Active Exploration for Inverse Reinforcement Learningcited by 27
  33. 2023When Demonstrations Meet Generative World Models: A Maximum Likelihood Framework for Offline Inverse Reinforcement Learningcited by 26
  34. 2024Hierarchical World Models as Visual Whole-Body Humanoid Controllerscited by 24
  35. 2018Deep Reinforcement Learning in Ice Hockey for Context-Aware Player Evaluationcited by 22
  36. 2023Towards Theoretical Understanding of Inverse Reinforcement Learningcited by 21
  37. 2023CFlowNets: Continuous Control with Generative Flow Networkscited by 19
  38. 2023Learning Safety Constraints from Demonstrations with Unknown Rewardscited by 19

50 most cited of 69 All 69 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 5; 2026: 5.

10 papers.

  1. 2025Modeling Others' Minds as Codecited by 6
  2. 2025Your Learned Constraint is Secretly a Backward Reachable Tubecited by 2
  3. 2025Explainable reinforcement learning from human feedback to improve alignmentcited by 1
  4. 2026Interactionless Inverse Reinforcement Learning: A Data-Centric Framework for Durable Alignmentcited by 1
  5. 2025A Domain-Based Taxonomy of Jailbreak Vulnerabilities in Large Language Modelscited by 0
  6. 2025DRIVE: Dynamic Rule Inference and Verified Evaluation for Constraint-Aware Autonomous Drivingcited by 0
  7. 2026Behavior Learning (BL): Learning Hierarchical Optimization Structures from Datacited by 0
  8. 2026Learning to maintain safety through expert demonstrations in settings with unknown constraints: A Q-learning perspectivecited by 0
  9. 2026The Alignment Flywheel: A Governance-Centric Hybrid MAS for Architecture-Agnostic Safetycited by 0
  10. 2026RS-Diffuser: Risk-Sensitive Diffusion Planning with Distributional Value Guidancecited by 0