mlsanityData to 14 Oct 2026

Massively Scalable Inverse Reinforcement Learning in Google Maps

arXiv 2305.11290 ↗First posted 18 May 2023cs.LGcited by 11

On the map: Offline RL and imitation learning from data → Inverse reinforcement learning

Abstract (arXiv)

Inverse reinforcement learning (IRL) offers a powerful and general framework for learning humans' latent preferences in route recommendation, yet no approach has successfully addressed planetary-scale problems with hundreds of millions of states and demonstration trajectories. In this paper, we introduce scaling techniques based on graph compression, spatial parallelization, and improved initialization conditions inspired by a connection to eigenvector algorithms. We revisit classic IRL methods in the routing context, and make the key observation that there exists a trade-off between the use of cheap, deterministic planners and expensive yet robust stochastic policies. This insight is leveraged in Receding Horizon Inverse Planning (RHIP), a new generalization of classic IRL algorithms that provides fine-grained control over performance trade-offs via its planning horizon. Our contributions culminate in a policy that achieves a 16-24% improvement in route quality at a global scale, and to the best of our knowledge, represents the largest published study of IRL algorithms in a real-world setting to date. We conclude by conducting an ablation study of key components, presenting negative results from alternative eigenvalue solvers, and identifying opportunities to further improve scalability via IRL-specific batching strategies.

Key facts. Cited by 11 papers in the map (data through 14 Oct 2026). Placed in Inverse reinforcement learning, one of 13 directions in Offline RL and imitation learning from data. Builds on 19 of its 61 references that are in the map. First posted on arXiv 18 May 2023.

Builds on

19 of 61 references.

Closest ancestors · 4

Same direction, newest first

  1. 2023Inverse Reinforcement Learning without Reinforcement Learning26 Mar
  2. 2021Scalable Bayesian Inverse Reinforcement Learning12 Feb
  3. 2019Deep Bayesian Reward Learning from Preferences10 Dec
  4. 2018Inverse reinforcement learning for video games24 Oct

Other references · 15

  1. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,509
  2. 2021Convex optimizationcited by 2,515
  3. 2016Generative Adversarial Imitation Learningcited by 1,597
  4. 2017Learning Robust Rewards with Adversarial Inverse Reinforcement Learningcited by 490
  5. 2018An Algorithmic Perspective on Imitation Learningcited by 459
  6. 2016Guided Cost Learning: Deep Inverse Optimal Control via Policy Optimizationcited by 406
  7. 2021IQ-Learn: Inverse soft-Q Learning for Imitationcited by 193
  8. 2019Imitation Learning via Off-Policy Distribution Matchingcited by 183
  9. 2015Maximum Entropy Deep Inverse Reinforcement Learningcited by 176
  10. 2016A Connection between Generative Adversarial Networks, Inverse Reinforcement Learning, and Energy-Based Modelscited by 168
  11. 2021ETA Prediction with Graph Neural Networks in Google Mapscited by 130
  12. 2019Imitation Learning as f-Divergence Minimizationcited by 123
3 more
  1. 2021Of Moments and Matching: A Game-Theoretic Framework for Closing the Imitation Gapcited by 81
  2. 2016Watch This: Scalable Cost-Function Learning for Path Planning in Urban Environmentscited by 33
  3. 2016Incorporating Human Domain Knowledge into Large Scale Cost Function Learningcited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 1; 2024: 5; 2025: 2; 2026: 3.

11 papers.

  1. 2024Imitating Language via Scalable Inverse Reinforcement Learningcited by 21
  2. 2024Diffusing States and Matching Scores: A New Framework for Imitation Learningcited by 16
  3. 2024Can Learned Optimization Make Reinforcement Learning Less Difficult?cited by 14
  4. 2024Multi-Agent Imitation Learning: Value is Easy, Regret is Hardcited by 11
  5. 2024EvIL: Evolution Strategies for Generalisable Imitation Learningcited by 8
  6. 2023Is Inverse Reinforcement Learning Harder than Standard Reinforcement Learning? A Theoretical Perspectivecited by 4
  7. 2025Model Predictive Adversarial Imitation Learning for Planning from Observationcited by 2
  8. 2025Value from Observations: Towards Large-Scale Imitation Learning via Self-Improvementcited by 0
  9. 2026Online World Modeling Enables Real-World Inverse Reinforcement Learning from Observationcited by 0
  10. 2026A Lecture Note on Offline RL and IRL, Part II: Foundations of Inverse Reinforcement Learning and Dynamic Discrete Choice Modelscited by 0
  11. 2026Q-based Variational Inverse Reinforcement Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2022Receding Horizon Inverse Reinforcement Learning
  2. 2024How does Inverse RL Scale to Large State Spaces? A Provably Efficient Approach
  3. 2018A Survey of Inverse Reinforcement Learning: Challenges, Methods and Progress
  4. 2025GoIRL: Graph-Oriented Inverse Reinforcement Learning for Multimodal Trajectory Prediction
  5. 2021A Generalised Inverse Reinforcement Learning Framework
  6. 2024Multi-Agent Inverse Reinforcement Learning in Real World Unstructured Pedestrian Crowds
  7. 2024Towards Sample-Efficiency and Generalization of Transfer and Inverse Reinforcement Learning: A Comprehensive Literature Review
  8. 2018A Framework and Method for Online Inverse Reinforcement Learning