mlsanityData to 7 Oct 2026

Offline Reinforcement Learning via Inverse Optimization

arXiv 2502.20030 ↗First posted 27 Feb 2025cs.LG cs.SY eess.SYcited by 2

On the map: Offline RL and imitation learning from data → Inverse reinforcement learning

Abstract (arXiv)

Inspired by the recent successes of Inverse Optimization (IO) across various application domains, we propose a novel offline Reinforcement Learning (ORL) algorithm for continuous state and action spaces, leveraging the convex loss function called ``sub-optimality loss'' from the IO literature. To mitigate the distribution shift commonly observed in ORL problems, we further employ a robust and non-causal Model Predictive Control (MPC) expert steering a nominal model of the dynamics using in-hindsight information stemming from the model mismatch. Unlike the existing literature, our robust MPC expert enjoys an exact and tractable convex reformulation. In the second part of this study, we show that the IO hypothesis class, trained by the proposed convex loss function, enjoys ample expressiveness and {reliably recovers teacher behavior in MuJoCo benchmarks. The method achieves competitive results compared to widely-used baselines in sample-constrained settings, despite using} orders of magnitude fewer parameters. To facilitate the reproducibility of our results, we provide an open-source package implementing the proposed algorithms and the experiments. The code is available at https://github.com/TolgaOk/offlineRLviaIO.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in Inverse reinforcement learning, one of 13 directions in Offline RL and imitation learning from data. Builds on 38 of its 54 references that are in the map. First posted on arXiv 27 Feb 2025.

Builds on

38 of 54 references.

References · 38

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2015Continuous control with deep reinforcement learningcited by 3,279
  3. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  4. 2021Convex optimizationcited by 2,479
  5. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,777
  6. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,406
  7. 2021Offline Reinforcement Learning with Implicit Q-Learningcited by 1,245
  8. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
  9. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 833
  10. 2019Behavior Regularized Offline Reinforcement Learningcited by 654
  11. 2020MOReL : Model-Based Offline Reinforcement Learningcited by 582
  12. 2021COMBO: Conservative Offline Model-Based Policy Optimizationcited by 365
26 more
  1. 2021Safe Learning in Robotics: From Learning-Based Control to Safe Reinforcement Learningcited by 337
  2. 2021Bridging Offline Reinforcement Learning and Imitation Learning: A Tale of Pessimismcited by 275
  3. 2019Batch Policy Learning under Constraintscited by 242
  4. 2021RvS: What is Essential for Offline RL via Supervised Learning?cited by 184
  5. 2017On the Sample Complexity of the Linear Quadratic Regulatorcited by 151
  6. 2020Naive Exploration is Optimal for Online LQRcited by 111
  7. 2018Plan Online, Learn Offline: Efficient Learning and Exploration via Model-Based Controlcited by 104
  8. 2019Online Control with Adversarial Disturbancescited by 92
  9. 2019BAIL: Best-Action Imitation Learning for Batch Deep Reinforcement Learningcited by 90
  10. 2019The Nonstochastic Control Problemcited by 52
  11. 2021Offline Reinforcement Learning with Reverse Model-based Imaginationcited by 47
  12. 2020Logarithmic Regret for Adversarial Online Controlcited by 41
  13. 2019Formulas for Data-driven Control: Stabilization, Optimality and Robustnesscited by 40
  14. 2024DiffStitch: Boosting Offline Reinforcement Learning with Diffusion-based Trajectory Stitchingcited by 35
  15. 2015Data-driven Inverse Optimization with Imperfect Informationcited by 33
  16. 2018Data-Enabled Predictive Control: In the Shallows of the DeePCcited by 33
  17. 2020Blending MPC & Value Function Approximation for Efficient Reinforcement Learningcited by 27
  18. 2024GTA: Generative Trajectory Augmentation with Guidance for Offline Reinforcement Learningcited by 22
  19. 2022Double Check Your State Before Trusting It: Confidence-Aware Bidirectional Offline Model-Based Imaginationcited by 21
  20. 2023Offline Primal-Dual Reinforcement Learning for Linear MDPscited by 17
  21. 2023Uncertainty-driven Trajectory Truncation for Data Augmentation in Offline Reinforcement Learningcited by 13
  22. 2023Learning in Inverse Optimization: Incenter Cost, Augmented Suboptimality Loss, and Algorithmscited by 10
  23. 2020From noisy data to feedback controllers: non-conservative design via a matrix S-lemmacited by 9
  24. 2023A Primal-Dual-Critic Algorithm for Offline Constrained Reinforcement Learningcited by 9
  25. 2020Regret-optimal measurement-feedback controlcited by 5
  26. 2024Scalable Kernel Inverse Optimizationcited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026Tight Generalization Bounds for Noiseless Inverse Optimizationcited by 4
  2. 2026Tight Regret Bound for Online Inverse Linear Optimization via Multiscale Matrix Weightscited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2023A Bayesian Approach to Robust Inverse Reinforcement Learning
  2. 2023Is Inverse Reinforcement Learning Harder than Standard Reinforcement Learning? A Theoretical Perspective
  3. 2024Offline Hierarchical Reinforcement Learning via Inverse Optimization
  4. 2025Inverse Reinforcement Learning via Convex Optimization
  5. 2018A Survey of Inverse Reinforcement Learning: Challenges, Methods and Progress
  6. 2025OffSim: Offline Simulator for Model-based Offline Inverse Reinforcement Learning
  7. 2021A Generalised Inverse Reinforcement Learning Framework
  8. 2023CLARE: Conservative Model-Based Reward Learning for Offline Inverse Reinforcement Learning