mlsanityData to 7 Oct 2026

Structural Estimation of Markov Decision Processes in High-Dimensional State Space with Finite-Time Guarantees

arXiv 2210.01282 ↗First posted 4 Oct 2022cs.LG cs.AI econ.EMcited by 12

On the map: Offline RL and imitation learning from data → Inverse reinforcement learning

Abstract (arXiv)

We consider the task of estimating a structural model of dynamic decisions by a human agent based upon the observable history of implemented actions and visited states. This problem has an inherent nested structure: in the inner problem, an optimal policy for a given reward function is identified while in the outer problem, a measure of fit is maximized. Several approaches have been proposed to alleviate the computational burden of this nested-loop structure, but these methods still suffer from high complexity when the state space is either discrete with large cardinality or continuous in high dimensions. Other approaches in the inverse reinforcement learning (IRL) literature emphasize policy estimation at the expense of reduced reward estimation accuracy. In this paper we propose a single-loop estimation algorithm with finite time guarantees that is equipped to deal with high-dimensional state spaces without compromising reward estimation accuracy. In the proposed algorithm, each policy improvement step is followed by a stochastic gradient step for likelihood maximization. We show that the proposed algorithm converges to a stationary solution with a finite-time guarantee. Further, if the reward is parameterized linearly, we show that the algorithm approximates the maximum likelihood estimator sublinearly. Finally, by using robotics control problems in MuJoCo and their transfer settings, we show that the proposed algorithm achieves superior performance compared with other IRL and imitation learning benchmarks.

Key facts. Cited by 12 papers in the map (data through 7 Oct 2026). Placed in Inverse reinforcement learning, one of 13 directions in Offline RL and imitation learning from data. Builds on 23 of its 53 references that are in the map. First posted on arXiv 4 Oct 2022.

Builds on

23 of 53 references.

Closest ancestors · 3

Same direction, newest first

  1. 2021Inverse Reinforcement Learning with Explicit Policy Estimates4 Mar
  2. 2020f-IRL: Inverse Reinforcement Learning via State Marginal Matching9 Nov
  3. 2020Robust Inverse Reinforcement Learning under Transition Dynamics Mismatch2 Jul

Other references · 20

  1. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  2. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  3. 2016Generative Adversarial Imitation Learningcited by 1,576
  4. 2020Deep Reinforcement Learning for Autonomous Driving: A Surveycited by 698
  5. 2017Reinforcement Learning with Deep Energy-Based Policiescited by 659
  6. 2018Gradient Descent Provably Optimizes Over-parameterized Neural Networkscited by 655
  7. 2017Learning Robust Rewards with Adversarial Inverse Reinforcement Learningcited by 485
  8. 2019Reinforcement Learning in Healthcare: A Surveycited by 308
  9. 2018A Finite Time Analysis of Temporal Difference Learning With Linear Function Approximationcited by 225
  10. 2021IQ-Learn: Inverse soft-Q Learning for Imitationcited by 191
  11. 2015Maximum Entropy Deep Inverse Reinforcement Learningcited by 173
  12. 2020Fast Global Convergence of Natural Policy Gradient Methods with Entropy Regularizationcited by 170
8 more
  1. 2020A Finite Time Analysis of Two Time-Scale Actor Critic Methodscited by 120
  2. 2021A Near-Optimal Algorithm for Stochastic Bilevel Optimization via Double-Momentumcited by 106
  3. 2019Finite-Sample Analysis for SARSA with Linear Function Approximationcited by 85
  4. 2019State Alignment-based Imitation Learningcited by 73
  5. 2020State-only Imitation with Transition Dynamics Mismatchcited by 37
  6. 2020On Computation and Generalization of Generative Adversarial Imitation Learningcited by 27
  7. 2020When Will Generative Adversarial Imitation Learning Algorithms Attain Global Convergencecited by 12
  8. 2020A Relation Analysis of Markov Decision Process Frameworkscited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 3; 2024: 4; 2025: 4; 2026: 1.

12 papers.

  1. 2023Reinforcement Learning with Human Feedback: Learning Dynamic Choices via Pessimismcited by 69
  2. 2024Getting More Juice Out of the SFT Data: Reward Learning from Human Demonstration Improves SFT for LLM Alignmentcited by 32
  3. 2023When Demonstrations Meet Generative World Models: A Maximum Likelihood Framework for Offline Inverse Reinforcement Learningcited by 26
  4. 2024Imitation Learning in Discounted Linear MDPs without exploration assumptionscited by 15
  5. 2025Optimistically Optimistic Exploration for Provably Efficient Infinite-Horizon Reinforcement and Imitation Learningcited by 12
  6. 2023A Bayesian Approach to Robust Inverse Reinforcement Learningcited by 9
  7. 2024A Unified Linear Programming Framework for Offline Reward Learning from Human Demonstrations and Feedbackcited by 4
  8. 2024Learning Reward and Policy Jointly from Demonstration and Preference Improves Alignmentcited by 4
  9. 2025From Demonstrations to Rewards: Alignment Without Explicit Human Preferencescited by 4
  10. 2025Stability and Generalization for Bellman Residualscited by 1
  11. 2025Recursive Deep Inverse Reinforcement Learningcited by 0
  12. 2026DeepHalo: A Neural Choice Model with Controllable Context Effectscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2022Maximum-Likelihood Inverse Reinforcement Learning with Finite-Time Guarantees
  2. 2024How does Inverse RL Scale to Large State Spaces? A Provably Efficient Approach
  3. 2023A Data-Driven State Aggregation Approach for Dynamic Discrete Choice Models
  4. 2019Maximum Likelihood Constraint Inference for Inverse Reinforcement Learning
  5. 2023FP-IRL: Fokker--Planck Inverse Reinforcement Learning -- A Physics-Constrained Approach to Markov Decision Processes
  6. 2026A Lecture Note on Offline RL and IRL, Part II: Foundations of Inverse Reinforcement Learning and Dynamic Discrete Choice Models
  7. 2023Model Selection for Inverse Reinforcement Learning via Structural Risk Minimization
  8. 2024Towards Generalized Inverse Reinforcement Learning