# CityFlow: A Multi-Agent Reinforcement Learning Environment for Large Scale City Traffic Scenario

arXiv: https://arxiv.org/abs/1905.05217 · HTML: https://mlsanity.com/paper/1905.05217/

## Key facts

- Cited by 115 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 13 May 2019.

## Built on by (most cited first, 30 of 115)

- [MetaDrive: Composing Diverse Driving Scenarios for Generalizable Reinforcement Learning](https://mlsanity.com/paper/2109.12674/) (2021)
- [AgentSociety: Large-Scale Simulation of LLM-Driven Generative Agents Advances Understanding of Human Behaviors and Society](https://mlsanity.com/paper/2502.08691/) (2025)
- [Embodied LLM Agents Learn to Cooperate in Organized Teams](https://mlsanity.com/paper/2403.12482/) (2024)
- [A Survey on Model-based Reinforcement Learning](https://mlsanity.com/paper/2206.09328/) (2022)
- [Open-TI: Open Traffic Intelligence with Augmented Language Model](https://mlsanity.com/paper/2401.00211/) (2023)
- [A Survey of Sim-to-Real Methods in RL: Progress, Prospects and Challenges with Foundation Models](https://mlsanity.com/paper/2502.13187/) (2025)
- [Prompt to Transfer: Sim-to-Real Transfer for Traffic Signal Control with Prompt Learning](https://mlsanity.com/paper/2308.14284/) (2023)
- [Learning to Simulate Self-Driven Particles System with Coordinated Policy Optimization](https://mlsanity.com/paper/2110.13827/) (2021)
- [Multi-Agent Reinforcement Learning for Autonomous Driving: A Survey](https://mlsanity.com/paper/2408.09675/) (2024)
- [Expression might be enough: representing pressure and demand for reinforcement learning based traffic signal control](https://mlsanity.com/paper/2112.10107/) (2021)
- [CityBench: Evaluating the Capabilities of Large Language Models for Urban Tasks](https://mlsanity.com/paper/2406.13945/) (2024)
- [Choose Your Simulator Wisely: A Review on Open-source Simulators for Autonomous Driving](https://mlsanity.com/paper/2311.11056/) (2023)
- [Efficient Pressure: Improving efficiency for signalized intersections](https://mlsanity.com/paper/2112.02336/) (2021)
- [An Introduction to Multi-Agent Reinforcement Learning and Review of its Application to Autonomous Mobility](https://mlsanity.com/paper/2203.07676/) (2022)
- [LibSignal: An Open Library for Traffic Signal Control](https://mlsanity.com/paper/2211.10649/) (2022)
- [CoLLMLight: Cooperative Large Language Model Agents for Network-Wide Traffic Signal Control](https://mlsanity.com/paper/2503.11739/) (2025)
- [Multi-Agent Reinforcement Learning for Resources Allocation Optimization: A Survey](https://mlsanity.com/paper/2504.21048/) (2025)
- [Reinforcement Learning Approaches for Traffic Signal Control under Missing Data](https://mlsanity.com/paper/2304.10722/) (2023)
- [FightLadder: A Benchmark for Competitive Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2406.02081/) (2024)
- [Modeling Network-level Traffic Flow Transitions on Sparse Data](https://mlsanity.com/paper/2208.06646/) (2022)
- [DynSTGAT: Dynamic Spatial-Temporal Graph Attention Network for Traffic Signal Control](https://mlsanity.com/paper/2109.05491/) (2021)
- [Multi-Agent Reinforcement Learning for Traffic Signal Control through Universal Communication Method](https://mlsanity.com/paper/2204.12190/) (2022)
- [The Real Deal: A Review of Challenges and Opportunities in Moving Reinforcement Learning-Based Traffic Signal Control Systems Towards Reality](https://mlsanity.com/paper/2206.11996/) (2022)
- [TransWorldNG: Traffic Simulation via Foundation Model](https://mlsanity.com/paper/2305.15743/) (2023)
- [IMP-MARL: a Suite of Environments for Large-scale Infrastructure Management Planning via MARL](https://mlsanity.com/paper/2306.11551/) (2023)
- [MetaVIM: Meta Variationally Intrinsic Motivated Reinforcement Learning for Decentralized Traffic Signal Control](https://mlsanity.com/paper/2101.00746/) (2021)
- [Objective-aware Traffic Simulation via Inverse Reinforcement Learning](https://mlsanity.com/paper/2105.09560/) (2021)
- [Consensus Learning for Cooperative Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2206.02583/) (2022)
- [Off-the-Grid MARL: Datasets with Baselines for Offline Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2302.00521/) (2023)
- [Space Meets Time: Local Spacetime Neural Network For Traffic Flow Forecasting](https://mlsanity.com/paper/2109.05225/) (2021)

## Abstract (arXiv)

Traffic signal control is an emerging application scenario for reinforcement learning. Besides being as an important problem that affects people's daily life in commuting, traffic signal control poses its unique challenges for reinforcement learning in terms of adapting to dynamic traffic environment and coordinating thousands of agents including vehicles and pedestrians. A key factor in the success of modern reinforcement learning relies on a good simulator to generate a large number of data samples for learning. The most commonly used open-source traffic simulator SUMO is, however, not scalable to large road network and large traffic flow, which hinders the study of reinforcement learning on traffic scenarios. This motivates us to create a new traffic simulator CityFlow with fundamentally optimized data structures and efficient algorithms. CityFlow can support flexible definitions for road network and traffic flow based on synthetic and real-world data. It also provides user-friendly interface for reinforcement learning. Most importantly, CityFlow is more than twenty times faster than SUMO and is capable of supporting city-wide traffic simulation with an interactive render for monitoring. Besides traffic signal control, CityFlow could serve as the base for other transportation studies and can create new possibilities to test machine learning methods in the intelligent transportation domain.