# SceneStreamer: Continuous Scenario Generation as Next Token Group Prediction

arXiv: https://arxiv.org/abs/2506.23316 · HTML: https://mlsanity.com/paper/2506.23316/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 56 of its 71 references that are in the map.
- First posted on arXiv 29 Jun 2025.

## Builds on (most cited first, 30 of 56)

- [PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation](https://mlsanity.com/paper/1612.00593/) (2016)
- [CARLA: An Open Urban Driving Simulator](https://mlsanity.com/paper/1711.03938/) (2017)
- [Addressing Function Approximation Error in Actor-Critic Methods](https://mlsanity.com/paper/1802.09477/) (2018)
- [FiLM: Visual Reasoning with a General Conditioning Layer](https://mlsanity.com/paper/1709.07871/) (2017)
- [MultiPath: Multiple Probabilistic Anchor Trajectory Hypotheses for Behavior Prediction](https://mlsanity.com/paper/1910.05449/) (2019)
- [MotionCLIP: Exposing Human Motion Generation to CLIP Space](https://mlsanity.com/paper/2203.08063/) (2022)
- [Vista: A Generalizable Driving World Model with High Fidelity and Versatile Controllability](https://mlsanity.com/paper/2405.17398/) (2024)
- [MetaDrive: Composing Diverse Driving Scenarios for Generalizable Reinforcement Learning](https://mlsanity.com/paper/2109.12674/) (2021)
- [Motion Transformer with Global Intention Localization and Local Movement Refinement](https://mlsanity.com/paper/2209.13508/) (2022)
- [MotionDiffuser: Controllable Multi-Agent Motion Prediction using Diffusion](https://mlsanity.com/paper/2306.03083/) (2023)
- [TrafficSim: Learning to Simulate Realistic Multi-Agent Behaviors](https://mlsanity.com/paper/2101.06557/) (2021)
- [Guided Conditional Diffusion for Controllable Traffic Simulation](https://mlsanity.com/paper/2210.17366/) (2022)
- [Waymax: An Accelerated, Data-Driven Simulator for Large-Scale Autonomous Driving Research](https://mlsanity.com/paper/2310.08710/) (2023)
- [MotionLM: Multi-Agent Motion Forecasting as Language Modeling](https://mlsanity.com/paper/2309.16534/) (2023)
- [MTR++: Multi-Agent Motion Prediction with Symmetric Scene Modeling and Guided Intention Querying](https://mlsanity.com/paper/2306.17770/) (2023)
- [Generating Useful Accident-Prone Driving Scenarios via a Learned Traffic Prior](https://mlsanity.com/paper/2112.05077/) (2021)
- [SMARTS: Scalable Multi-Agent Reinforcement Learning Training School for Autonomous Driving](https://mlsanity.com/paper/2010.09776/) (2020)
- [Language-Guided Traffic Simulation via Scene-Level Diffusion](https://mlsanity.com/paper/2306.06344/) (2023)
- [TrafficGen: Learning to Generate Diverse and Realistic Traffic Scenarios](https://mlsanity.com/paper/2210.06609/) (2022)
- [UniScene: Unified Occupancy-centric Driving Scene Generation](https://mlsanity.com/paper/2412.05435/) (2024)
- [DriveArena: A Closed-loop Generative Simulation Platform for Autonomous Driving](https://mlsanity.com/paper/2408.00415/) (2024)
- [Language Conditioned Traffic Generation](https://mlsanity.com/paper/2307.07947/) (2023)
- [SceneGen: Learning to Generate Realistic Traffic Scenes](https://mlsanity.com/paper/2101.06541/) (2021)
- [SimNet: Learning Reactive Self-driving Simulations from Real-world Observations](https://mlsanity.com/paper/2105.12332/) (2021)
- [ScenarioNet: Open-Source Platform for Large-Scale Traffic Scenario Simulation and Modeling](https://mlsanity.com/paper/2306.12241/) (2023)
- [Trajeglish: Traffic Modeling as Next-Token Prediction](https://mlsanity.com/paper/2312.04535/) (2023)
- [Nocturne: a scalable driving benchmark for bringing multi-agent learning one step closer to the real world](https://mlsanity.com/paper/2206.09889/) (2022)
- [TrafficBots: Towards World Models for Autonomous Driving Simulation and Motion Prediction](https://mlsanity.com/paper/2303.04116/) (2023)
- [CAT: Closed-loop Adversarial Training for Safe End-to-End Driving](https://mlsanity.com/paper/2310.12432/) (2023)
- [BehaviorGPT: Smart Agent Simulation for Autonomous Driving with Next-Patch Prediction](https://mlsanity.com/paper/2405.17372/) (2024)

## Built on by (most cited first, 2 of 2)

- [DecompGAIL: Learning Realistic Traffic Behaviors with Decomposed Multi-Agent Generative Adversarial Imitation Learning](https://mlsanity.com/paper/2510.06913/) (2025)
- [AutoWorld: Learning Multi-Agent Traffic Simulation with Self-Supervised World Models](https://mlsanity.com/paper/2603.28963/) (2026)

## Abstract (arXiv)

Realistic and interactive traffic simulation is essential for training and evaluating autonomous driving systems. However, most existing data-driven simulation methods rely on static initialization or log-replay data, limiting their ability to model dynamic, long-horizon scenarios with evolving agent populations. We propose SceneStreamer, a unified autoregressive framework for continuous scenario generation that represents the entire scene as a sequence of tokens, including traffic light signals, agent states, and motion vectors, and generates them step by step with a transformer model. This design enables SceneStreamer to continuously introduce and retire agents over an unbounded horizon, supporting realistic long-duration simulation. Experiments demonstrate that SceneStreamer produces realistic, diverse, and adaptive traffic behaviors. Furthermore, reinforcement learning policies trained in SceneStreamer-generated scenarios achieve superior robustness and generalization, validating its utility as a high-fidelity simulation environment for autonomous driving. More information is available at https://vail-ucla.github.io/scenestreamer/ .