Understanding Inverse Reinforcement Learning under Overparameterization: Non-Asymptotic Analysis and Global Optimality
On the map: Offline RL and imitation learning from data → Inverse reinforcement learning
Abstract (arXiv)
The goal of the Inverse reinforcement learning (IRL) task is to identify the underlying reward function and the corresponding optimal policy from a set of expert demonstrations. While most IRL algorithms' theoretical guarantees rely on a linear reward structure, we aim to extend the theoretical understanding of IRL to scenarios where the reward function is parameterized by neural networks. Meanwhile, conventional IRL algorithms usually adopt a nested structure, leading to computational inefficiency, especially in high-dimensional settings. To address this problem, we propose the first two-timescale single-loop IRL algorithm under neural network parameterized reward and provide a non-asymptotic convergence analysis under overparameterization. Although prior optimality results for linear rewards do not apply, we show that our algorithm can identify the globally optimal reward and policy under certain neural network structures. This is the first IRL algorithm with a non-asymptotic convergence guarantee that provably achieves global optimality in neural network settings.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Inverse reinforcement learning, one of 13 directions in Offline RL and imitation learning from data. Builds on 37 of its 59 references that are in the map. First posted on arXiv 22 Mar 2025.
Builds on
37 of 59 references.
Closest ancestors · 3
Same direction, newest first
- 2023When Demonstrations Meet Generative World Models: A Maximum Likelihood Framework for Offline Inverse Reinforcement Learning15 Feb
- 2022Maximum-Likelihood Inverse Reinforcement Learning with Finite-Time Guarantees4 Oct
- 2020f-IRL: Inverse Reinforcement Learning via State Marginal Matching9 Nov
Other references · 34
- 2022Training language models to follow instructions with human feedbackcited by 15,971
- 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
- 2016Generative Adversarial Imitation Learningcited by 1,597
- 2018A Convergence Theory for Deep Learning via Over-Parameterizationcited by 787
- 2017Reinforcement Learning with Deep Energy-Based Policiescited by 669
- 2019Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networkscited by 566
- 2018An Algorithmic Perspective on Imitation Learningcited by 459
- 2018Lipschitz regularity of deep neural networks: analysis and efficient estimationcited by 415
- 2016Guided Cost Learning: Deep Inverse Optimal Control via Policy Optimizationcited by 406
- 2018Learning and Generalization in Overparameterized Neural Networks, Going Beyond Two Layerscited by 340
- 2021Bridging Offline Reinforcement Learning and Imitation Learning: A Tale of Pessimismcited by 279
- 2018Learning Overparameterized Neural Networks via Stochastic Gradient Descent on Structured Datacited by 261
22 more
- 2020Bilevel Optimization: Convergence Analysis and Enhanced Designcited by 222
- 2021IQ-Learn: Inverse soft-Q Learning for Imitationcited by 193
- 2015Maximum Entropy Deep Inverse Reinforcement Learningcited by 176
- 2020Fast Global Convergence of Natural Policy Gradient Methods with Entropy Regularizationcited by 176
- 2019Neural Policy Gradient Methods: Global Optimality and Rates of Convergencecited by 159
- 2018Stochastic Gradient Descent Optimizes Over-parameterized Deep ReLU Networkscited by 141
- 2020A Finite Time Analysis of Two Time-Scale Actor Critic Methodscited by 122
- 2021A Near-Optimal Algorithm for Stochastic Bilevel Optimization via Double-Momentumcited by 106
- 2018On the Power of Over-parametrization in Neural Networks with Quadratic Activationcited by 102
- 2020Toward the Fundamental Limits of Imitation Learningcited by 96
- 2019Finite-Sample Analysis for SARSA with Linear Function Approximationcited by 85
- 2019State Alignment-based Imitation Learningcited by 73
- 2019Convergence of Adversarial Training in Overparametrized Neural Networkscited by 58
- 2017SGD Learns the Conjugate Kernel Class of the Networkcited by 54
- 2020Single-Timescale Actor-Critic Provably Finds Globally Optimal Policycited by 34
- 2018A Large-Scale Study on Regularization and Normalization in GANscited by 26
- 2019Neural Temporal-Difference and Q-Learning Provably Converge to Global Optimacited by 20
- 2022Driving in Real Life with Inverse Reinforcement Learningcited by 18
- 2020When Will Generative Adversarial Imitation Learning Algorithms Attain Global Convergencecited by 13
- 2018Feature-Based Aggregation and Deep Reinforcement Learning: A Survey and Some New Implementationscited by 12
- 2020Generative Adversarial Imitation Learning with Neural Networks: Global Optimality and Convergence Ratecited by 7
- 2023On the Global Convergence of Natural Actor-Critic with Two-layer Neural Network Parametrizationcited by 5
Built on by
Citing papers per year: 2026: 1.
1 paper.
Similar, not cited
Close in wording, no citation either way
- 2018A Survey of Inverse Reinforcement Learning: Challenges, Methods and Progress
- 2020Inverse Reinforcement Learning via Matching of Optimality Profiles
- 2026Inverse Reinforcement Learning without an Optimal Demonstrator: A Feasible Reward Set Approach
- 2023A Novel Variational Lower Bound for Inverse Reinforcement Learning
- 2023Is Inverse Reinforcement Learning Harder than Standard Reinforcement Learning? A Theoretical Perspective
- 2023BC-IRL: Learning Generalizable Reward Functions from Demonstrations
- 2023Towards Theoretical Understanding of Inverse Reinforcement Learning
- 2024Bayesian Inverse Reinforcement Learning for Non-Markovian Rewards