# Single Agent Robust Deep Reinforcement Learning for Bus Fleet Control

arXiv: https://arxiv.org/abs/2508.20784 · HTML: https://mlsanity.com/paper/2508.20784/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Placed in Multi-agent reinforcement learning, one of 15 directions in Deep reinforcement learning methods.
- Builds on 6 of its 28 references that are in the map.
- First posted on arXiv 28 Aug 2025.

## Placed in

- Deep reinforcement learning methods / [Multi-agent reinforcement learning](https://mlsanity.com/deeprl/multi-agent-reinforcement-learning/)

## Builds on (most cited first, 6 of 6)

- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [Soft Actor-Critic Algorithms and Applications](https://mlsanity.com/paper/1812.05905/) (2018)
- [Maximum Entropy RL (Provably) Solves Some Robust RL Problems](https://mlsanity.com/paper/2103.06257/) (2021)
- [Deep Reinforcement Learning for Event-Driven Multi-Agent Decision Processes](https://mlsanity.com/paper/1709.06656/) (2017)
- [Reducing Bus Bunching with Asynchronous Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2105.00376/) (2021)
- [A Robust Integrated Multi-Strategy Bus Control System via Deep Reinforcement Learning](https://mlsanity.com/paper/2308.08179/) (2023)

## Built on by (most cited first, 4 of 4)

- [RE-SAC: Disentangling aleatoric and epistemic risks in bus fleet control: A stable and robust ensemble DRL approach](https://mlsanity.com/paper/2603.18396/) (2026)
- [BAPR: Bayesian amnesic piecewise-robust reinforcement learning for non-stationary continuous control](https://mlsanity.com/paper/2605.16170/) (2026)
- [Mitigating Bus Bunching with Reinforcement Learning Enhanced by Semantic Stop Embedding](https://mlsanity.com/paper/2608.10207/) (2026)
- [Arithmetic Actor Heads and Training Stabilization for Out-of-Distribution Reinforcement Learning](https://mlsanity.com/paper/2610.05143/) (2026)

## Abstract (arXiv)

Bus bunching remains a challenge for urban transit due to stochastic traffic and passenger demand. Traditional solutions rely on multi-agent reinforcement learning (MARL) in loop-line settings, which overlook realistic operations characterized by heterogeneous routes, timetables, fluctuating demand, and varying fleet sizes. We propose a novel single-agent reinforcement learning (RL) framework for bus holding control that avoids the data imbalance and convergence issues of MARL under near-realistic simulation. A bidirectional timetabled network with dynamic passenger demand is constructed. The key innovation is reformulating the multi-agent problem into a single-agent one by augmenting the state space with categorical identifiers (vehicle ID, station ID, time period) in addition to numerical features (headway, occupancy, velocity). This high-dimensional encoding enables single-agent policies to capture inter-agent dependencies, analogous to projecting non-separable inputs into a higher-dimensional space. We further design a structured reward function aligned with operational goals: instead of exponential penalties on headway deviations, a ridge-shaped reward balances uniform headways and schedule adherence. Experiments show that our modified soft actor-critic (SAC) achieves more stable and superior performance than benchmarks, including MADDPG (e.g., -430k vs. -530k under stochastic conditions). These results demonstrate that single-agent deep RL, when enhanced with categorical structuring and schedule-aware rewards, can effectively manage bus holding in non-loop, real-world contexts. This paradigm offers a robust, scalable alternative to MARL frameworks, particularly where agent-specific experiences are imbalanced.