# Learning to Control Autonomous Fleets from Observation via Offline Reinforcement Learning

arXiv: https://arxiv.org/abs/2302.14833 · HTML: https://mlsanity.com/paper/2302.14833/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in Imitation and offline RL for driving, one of 13 directions in Offline RL and imitation learning from data.
- Builds on 36 of its 69 references that are in the map.
- First posted on arXiv 28 Feb 2023.

## Placed in

- Offline RL and imitation learning from data / [Imitation and offline RL for driving](https://mlsanity.com/offlinerl/imitation-and-offline-rl-for-driving/)

## Builds on (most cited first, 30 of 36)

- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems](https://mlsanity.com/paper/2005.01643/) (2020)
- [Conservative Q-Learning for Offline Reinforcement Learning](https://mlsanity.com/paper/2006.04779/) (2020)
- [Decision Transformer: Reinforcement Learning via Sequence Modeling](https://mlsanity.com/paper/2106.01345/) (2021)
- [D4RL: Datasets for Deep Data-Driven Reinforcement Learning](https://mlsanity.com/paper/2004.07219/) (2020)
- [Off-Policy Deep Reinforcement Learning without Exploration](https://mlsanity.com/paper/1812.02900/) (2018)
- [Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reduction](https://mlsanity.com/paper/1906.00949/) (2019)
- [Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learning](https://mlsanity.com/paper/1910.00177/) (2019)
- [QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation](https://mlsanity.com/paper/1806.10293/) (2018)
- [Offline Reinforcement Learning as One Big Sequence Modeling Problem](https://mlsanity.com/paper/2106.02039/) (2021)
- [MOPO: Model-based Offline Policy Optimization](https://mlsanity.com/paper/2005.13239/) (2020)
- [MOReL : Model-Based Offline Reinforcement Learning](https://mlsanity.com/paper/2005.05951/) (2020)
- [A Survey on Offline Reinforcement Learning: Taxonomy, Review, and Open Problems](https://mlsanity.com/paper/2203.01387/) (2022)
- [Closing the Sim-to-Real Loop: Adapting Simulation Randomization with Real World Experience](https://mlsanity.com/paper/1810.05687/) (2018)
- [Offline-to-Online Reinforcement Learning via Balanced Replay and Pessimistic Q-Ensemble](https://mlsanity.com/paper/2107.00591/) (2021)
- [Empirical Study of Off-Policy Policy Evaluation for Reinforcement Learning](https://mlsanity.com/paper/1911.06854/) (2019)
- [Supervised Reinforcement Learning with Recurrent Neural Network for Dynamic Treatment Recommendation](https://mlsanity.com/paper/1807.01473/) (2018)
- [Offline A/B testing for Recommender Systems](https://mlsanity.com/paper/1801.07030/) (2018)
- [A Workflow for Offline Model-Free Robotic Reinforcement Learning](https://mlsanity.com/paper/2109.10813/) (2021)
- [NeoRL: A Near Real-World Benchmark for Offline Reinforcement Learning](https://mlsanity.com/paper/2102.00714/) (2021)
- [Reinforcement Learning for Ridesharing: An Extended Survey](https://mlsanity.com/paper/2105.01099/) (2021)
- [Deep Reinforcement Learning for Multi-Driver Vehicle Dispatching and Repositioning Problem](https://mlsanity.com/paper/1911.11260/) (2019)
- [Offline Reinforcement Learning for Autonomous Driving with Safety and Exploration Enhancement](https://mlsanity.com/paper/2110.07067/) (2021)
- [Graph Neural Network Reinforcement Learning for Autonomous Mobility-on-Demand Systems](https://mlsanity.com/paper/2104.11434/) (2021)
- [Control of Robotic Mobility-On-Demand Systems: a Queueing-Theoretical Perspective](https://mlsanity.com/paper/1404.4391/) (2014)
- [Data-Driven Model Predictive Control of Autonomous Mobility-on-Demand Systems](https://mlsanity.com/paper/1709.07032/) (2017)
- [Real-world Ride-hailing Vehicle Repositioning using Deep Reinforcement Learning](https://mlsanity.com/paper/2103.04555/) (2021)
- [Transferring Multi-Agent Reinforcement Learning Policies for Autonomous Driving using Sim-to-Real](https://mlsanity.com/paper/2203.11653/) (2022)
- [Value Function is All You Need: A Unified Learning Framework for Ride Hailing Platforms](https://mlsanity.com/paper/2105.08791/) (2021)

## Built on by (most cited first, 2 of 2)

- [Atomic Proximal Policy Optimization for Electric Robo-Taxi Dispatch and Charger Allocation](https://mlsanity.com/paper/2502.13392/) (2025)
- [Offline Reinforcement Learning for Microgrid Voltage Regulation](https://mlsanity.com/paper/2505.09920/) (2025)

## Abstract (arXiv)

Autonomous Mobility-on-Demand (AMoD) systems are an evolving mode of transportation in which a centrally coordinated fleet of self-driving vehicles dynamically serves travel requests. The control of these systems is typically formulated as a large network optimization problem, and reinforcement learning (RL) has recently emerged as a promising approach to solve the open challenges in this space. Recent centralized RL approaches focus on learning from online data, ignoring the per-sample-cost of interactions within real-world transportation systems. To address these limitations, we propose to formalize the control of AMoD systems through the lens of offline reinforcement learning and learn effective control strategies using solely offline data, which is readily available to current mobility operators. We further investigate design decisions and provide empirical evidence based on data from real-world mobility systems showing how offline learning allows to recover AMoD control policies that (i) exhibit performance on par with online methods, (ii) allow for sample-efficient online fine-tuning and (iii) eliminate the need for complex simulation environments. Crucially, this paper demonstrates that offline RL is a promising paradigm for the application of RL-based solutions within economically-critical systems, such as mobility systems.