# Massively Scalable Inverse Reinforcement Learning in Google Maps

arXiv: https://arxiv.org/abs/2305.11290 · HTML: https://mlsanity.com/paper/2305.11290/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Placed in Inverse reinforcement learning, one of 13 directions in Offline RL and imitation learning from data.
- Builds on 19 of its 61 references that are in the map.
- First posted on arXiv 18 May 2023.

## Placed in

- Offline RL and imitation learning from data / [Inverse reinforcement learning](https://mlsanity.com/offlinerl/inverse-reinforcement-learning/)

## Builds on (most cited first, 19 of 19)

- [Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer](https://mlsanity.com/paper/1701.06538/) (2017)
- [Convex optimization](https://mlsanity.com/paper/2106.01946/) (2021)
- [Generative Adversarial Imitation Learning](https://mlsanity.com/paper/1606.03476/) (2016)
- [Learning Robust Rewards with Adversarial Inverse Reinforcement Learning](https://mlsanity.com/paper/1710.11248/) (2017)
- [An Algorithmic Perspective on Imitation Learning](https://mlsanity.com/paper/1811.06711/) (2018)
- [Guided Cost Learning: Deep Inverse Optimal Control via Policy Optimization](https://mlsanity.com/paper/1603.00448/) (2016)
- [IQ-Learn: Inverse soft-Q Learning for Imitation](https://mlsanity.com/paper/2106.12142/) (2021)
- [Imitation Learning via Off-Policy Distribution Matching](https://mlsanity.com/paper/1912.05032/) (2019)
- [Maximum Entropy Deep Inverse Reinforcement Learning](https://mlsanity.com/paper/1507.04888/) (2015)
- [A Connection between Generative Adversarial Networks, Inverse Reinforcement Learning, and Energy-Based Models](https://mlsanity.com/paper/1611.03852/) (2016)
- [ETA Prediction with Graph Neural Networks in Google Maps](https://mlsanity.com/paper/2108.11482/) (2021)
- [Imitation Learning as f-Divergence Minimization](https://mlsanity.com/paper/1905.12888/) (2019)
- [Of Moments and Matching: A Game-Theoretic Framework for Closing the Imitation Gap](https://mlsanity.com/paper/2103.03236/) (2021)
- [Scalable Bayesian Inverse Reinforcement Learning](https://mlsanity.com/paper/2102.06483/) (2021)
- [Inverse Reinforcement Learning without Reinforcement Learning](https://mlsanity.com/paper/2303.14623/) (2023)
- [Watch This: Scalable Cost-Function Learning for Path Planning in Urban Environments](https://mlsanity.com/paper/1607.02329/) (2016)
- [Deep Bayesian Reward Learning from Preferences](https://mlsanity.com/paper/1912.04472/) (2019)
- [Inverse reinforcement learning for video games](https://mlsanity.com/paper/1810.10593/) (2018)
- [Incorporating Human Domain Knowledge into Large Scale Cost Function Learning](https://mlsanity.com/paper/1612.04318/) (2016)

## Built on by (most cited first, 11 of 11)

- [Imitating Language via Scalable Inverse Reinforcement Learning](https://mlsanity.com/paper/2409.01369/) (2024)
- [Diffusing States and Matching Scores: A New Framework for Imitation Learning](https://mlsanity.com/paper/2410.13855/) (2024)
- [Can Learned Optimization Make Reinforcement Learning Less Difficult?](https://mlsanity.com/paper/2407.07082/) (2024)
- [Multi-Agent Imitation Learning: Value is Easy, Regret is Hard](https://mlsanity.com/paper/2406.04219/) (2024)
- [EvIL: Evolution Strategies for Generalisable Imitation Learning](https://mlsanity.com/paper/2406.11905/) (2024)
- [Is Inverse Reinforcement Learning Harder than Standard Reinforcement Learning? A Theoretical Perspective](https://mlsanity.com/paper/2312.00054/) (2023)
- [Model Predictive Adversarial Imitation Learning for Planning from Observation](https://mlsanity.com/paper/2507.21533/) (2025)
- [Value from Observations: Towards Large-Scale Imitation Learning via Self-Improvement](https://mlsanity.com/paper/2507.06701/) (2025)
- [Online World Modeling Enables Real-World Inverse Reinforcement Learning from Observation](https://mlsanity.com/paper/2602.24121/) (2026)
- [A Lecture Note on Offline RL and IRL, Part II: Foundations of Inverse Reinforcement Learning and Dynamic Discrete Choice Models](https://mlsanity.com/paper/2605.30843/) (2026)
- [Q-based Variational Inverse Reinforcement Learning](https://mlsanity.com/paper/2608.16888/) (2026)

## Abstract (arXiv)

Inverse reinforcement learning (IRL) offers a powerful and general framework for learning humans' latent preferences in route recommendation, yet no approach has successfully addressed planetary-scale problems with hundreds of millions of states and demonstration trajectories. In this paper, we introduce scaling techniques based on graph compression, spatial parallelization, and improved initialization conditions inspired by a connection to eigenvector algorithms. We revisit classic IRL methods in the routing context, and make the key observation that there exists a trade-off between the use of cheap, deterministic planners and expensive yet robust stochastic policies. This insight is leveraged in Receding Horizon Inverse Planning (RHIP), a new generalization of classic IRL algorithms that provides fine-grained control over performance trade-offs via its planning horizon. Our contributions culminate in a policy that achieves a 16-24% improvement in route quality at a global scale, and to the best of our knowledge, represents the largest published study of IRL algorithms in a real-world setting to date. We conclude by conducting an ablation study of key components, presenting negative results from alternative eigenvalue solvers, and identifying opportunities to further improve scalability via IRL-specific batching strategies.