# Coordinating Ride-Pooling with Public Transit using Reward-Guided Conservative Q-Learning: An Offline Training and Online Fine-Tuning Reinforcement Learning Framework

arXiv: https://arxiv.org/abs/2501.14199 · HTML: https://mlsanity.com/paper/2501.14199/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Placed in Imitation and offline RL for driving, one of 13 directions in Offline RL and imitation learning from data.
- Builds on 26 of its 57 references that are in the map.
- First posted on arXiv 24 Jan 2025.

## Placed in

- Offline RL and imitation learning from data / [Imitation and offline RL for driving](https://mlsanity.com/offlinerl/imitation-and-offline-rl-for-driving/)

## Builds on (most cited first, 26 of 26)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [Addressing Function Approximation Error in Actor-Critic Methods](https://mlsanity.com/paper/1802.09477/) (2018)
- [Conservative Q-Learning for Offline Reinforcement Learning](https://mlsanity.com/paper/2006.04779/) (2020)
- [Deep Reinforcement Learning with Double Q-learning](https://mlsanity.com/paper/1509.06461/) (2015)
- [Off-Policy Deep Reinforcement Learning without Exploration](https://mlsanity.com/paper/1812.02900/) (2018)
- [A Minimalist Approach to Offline Reinforcement Learning](https://mlsanity.com/paper/2106.06860/) (2021)
- [Cal-QL: Calibrated Offline RL Pre-Training for Efficient Online Fine-Tuning](https://mlsanity.com/paper/2303.05479/) (2023)
- [Offline-to-Online Reinforcement Learning via Balanced Replay and Pessimistic Q-Ensemble](https://mlsanity.com/paper/2107.00591/) (2021)
- [Q-Transformer: Scalable Offline Reinforcement Learning via Autoregressive Q-Functions](https://mlsanity.com/paper/2309.10150/) (2023)
- [DeepPool: Distributed Model-free Algorithm for Ride-sharing using Deep Reinforcement Learning](https://mlsanity.com/paper/1903.03882/) (2019)
- [A Deep Value-network Based Approach for Multi-Driver Order Dispatching](https://mlsanity.com/paper/2106.04493/) (2021)
- [Neural Approximate Dynamic Programming for On-Demand Ride-Pooling](https://mlsanity.com/paper/1911.08842/) (2019)
- [MOVI: A Model-Free Approach to Dynamic Fleet Management](https://mlsanity.com/paper/1804.04758/) (2018)
- [Real-world Ride-hailing Vehicle Repositioning using Deep Reinforcement Learning](https://mlsanity.com/paper/2103.04555/) (2021)
- [Hybrid Multi-agent Deep Reinforcement Learning for Autonomous Mobility on Demand Systems](https://mlsanity.com/paper/2212.07313/) (2022)
- [A Distributed Model-Free Ride-Sharing Approach for Joint Matching, Pricing, and Dispatching using Deep Reinforcement Learning](https://mlsanity.com/paper/2010.01755/) (2020)
- [Value Function is All You Need: A Unified Learning Framework for Ride Hailing Platforms](https://mlsanity.com/paper/2105.08791/) (2021)
- [A Distributed Model-Free Algorithm for Multi-hop Ride-sharing using Deep Reinforcement Learning](https://mlsanity.com/paper/1910.14002/) (2019)
- [Reinforcement Learning in the Wild: Scalable RL Dispatching Algorithm Deployed in Ridehailing Marketplace](https://mlsanity.com/paper/2202.05118/) (2022)
- [Real-time City-scale Ridesharing via Linear Assignment Problems](https://mlsanity.com/paper/1902.10676/) (2019)
- [A dynamic ridesharing dispatch and idle vehicle repositioning strategy with integrated transit transfers](https://mlsanity.com/paper/1901.00760/) (2018)
- [AdaPool: A Diurnal-Adaptive Fleet Management Framework using Model-Free Deep Reinforcement Learning and Change Point Detection](https://mlsanity.com/paper/2104.00203/) (2021)
- [Regulating For-Hire Autonomous Vehicles for An Equitable Multimodal Transportation Network](https://mlsanity.com/paper/2301.05798/) (2023)
- [Algorithms and Computational Study on a Transportation System Integrating Public Transit and Ridesharing of Personal Vehicles](https://mlsanity.com/paper/2308.09191/) (2023)

## Built on by (most cited first, 3 of 3)

- [One Step is Enough: Multi-Agent Reinforcement Learning based on One-Step Policy Optimization for Order Dispatch on Ride-Sharing Platforms](https://mlsanity.com/paper/2507.15351/) (2025)
- [Triple-BERT: Do We Really Need MARL for Order Dispatch on Ride-Sharing Platforms?](https://mlsanity.com/paper/2510.03257/) (2025)
- [Completion-Aware Cross-Fidelity Offline-to-Online Reinforcement Learning for Multi-Line Bus Holding](https://mlsanity.com/paper/2609.39868/) (2026)

## Abstract (arXiv)

This paper introduces a novel reinforcement learning (RL) framework, termed Reward-Guided Conservative Q-learning (RG-CQL), to enhance coordination between ride-pooling and public transit within a multimodal transportation network. We model each ride-pooling vehicle as an agent governed by a Markov Decision Process (MDP) and propose an offline training and online fine-tuning RL framework to learn the optimal operational decisions of the multimodal transportation systems, including rider-vehicle matching, selection of drop-off locations for passengers, and vehicle routing decisions, with improved data efficiency. During the offline training phase, we develop a Conservative Double Deep Q Network (CDDQN) as the action executor and a supervised learning-based reward estimator, termed the Guider Network, to extract valuable insights into action-reward relationships from data batches. In the online fine-tuning phase, the Guider Network serves as an exploration guide, aiding CDDQN in effectively and conservatively exploring unknown state-action pairs. The efficacy of our algorithm is demonstrated through a realistic case study using real-world data from Manhattan. We show that integrating ride-pooling with public transit outperforms two benchmark cases solo rides coordinated with transit and ride-pooling without transit coordination by 17% and 22% in the achieved system rewards, respectively. Furthermore, our innovative offline training and online fine-tuning framework offers a remarkable 81.3% improvement in data efficiency compared to traditional online RL methods with adequate exploration budgets, with a 4.3% increase in total rewards and a 5.6% reduction in overestimation errors. Experimental results further demonstrate that RG-CQL effectively addresses the challenges of transitioning from offline to online RL in large-scale ride-pooling systems integrated with transit.