# Trajeglish: Traffic Modeling as Next-Token Prediction

arXiv: https://arxiv.org/abs/2312.04535 · HTML: https://mlsanity.com/paper/2312.04535/

## Key facts

- Cited by 60 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 29 references that are in the map.
- First posted on arXiv 7 Dec 2023.

## Builds on (most cited first, 23 of 23)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [nuScenes: A multimodal dataset for autonomous driving](https://mlsanity.com/paper/1903.11027/) (2019)
- [FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness](https://mlsanity.com/paper/2205.14135/) (2022)
- [The Curious Case of Neural Text Degeneration](https://mlsanity.com/paper/1904.09751/) (2019)
- [Decoupled Weight Decay Regularization](https://mlsanity.com/paper/1711.05101/) (2017)
- [WaveNet: A Generative Model for Raw Audio](https://mlsanity.com/paper/1609.03499/) (2016)
- [Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Books](https://mlsanity.com/paper/1506.06724/) (2015)
- [Offline Reinforcement Learning as One Big Sequence Modeling Problem](https://mlsanity.com/paper/2106.02039/) (2021)
- [Sequence Level Training with Recurrent Neural Networks](https://mlsanity.com/paper/1511.06732/) (2015)
- [Large Scale Interactive Motion Forecasting for Autonomous Driving : The Waymo Open Motion Dataset](https://mlsanity.com/paper/2104.10133/) (2021)
- [VectorNet: Encoding HD Maps and Agent Dynamics from Vectorized Representation](https://mlsanity.com/paper/2005.04259/) (2020)
- [Using the Output Embedding to Improve Language Models](https://mlsanity.com/paper/1608.05859/) (2016)
- [Motion Transformer with Global Intention Localization and Local Movement Refinement](https://mlsanity.com/paper/2209.13508/) (2022)
- [MultiPath++: Efficient Information Fusion and Trajectory Aggregation for Behavior Prediction](https://mlsanity.com/paper/2111.14973/) (2021)
- [TrafficSim: Learning to Simulate Realistic Multi-Agent Behaviors](https://mlsanity.com/paper/2101.06557/) (2021)
- [Guided Conditional Diffusion for Controllable Traffic Simulation](https://mlsanity.com/paper/2210.17366/) (2022)
- [MTR++: Multi-Agent Motion Prediction with Symmetric Scene Modeling and Guided Intention Querying](https://mlsanity.com/paper/2306.17770/) (2023)
- [Generating Useful Accident-Prone Driving Scenarios via a Learned Traffic Prior](https://mlsanity.com/paper/2112.05077/) (2021)
- [Imitation Is Not Enough: Robustifying Imitation with Reinforcement Learning for Challenging Driving Scenarios](https://mlsanity.com/paper/2212.11419/) (2022)
- [FastDraw: Addressing the Long Tail of Lane Detection by Adapting a Sequential Prediction Network](https://mlsanity.com/paper/1905.04354/) (2019)

## Built on by (most cited first, 30 of 60)

- [VADv2: End-to-End Vectorized Autonomous Driving via Probabilistic Planning](https://mlsanity.com/paper/2402.13243/) (2024)
- [AutoVLA: A Vision-Language-Action Model for End-to-End Autonomous Driving with Adaptive Reasoning and Reinforcement Fine-Tuning](https://mlsanity.com/paper/2506.13757/) (2025)
- [BehaviorGPT: Smart Agent Simulation for Autonomous Driving with Next-Patch Prediction](https://mlsanity.com/paper/2405.17372/) (2024)
- [Closed-Loop Supervised Fine-Tuning of Tokenized Traffic Models](https://mlsanity.com/paper/2412.05334/) (2024)
- [SAFE-SIM: Safety-Critical Closed-Loop Traffic Simulation with Diffusion-Controllable Adversaries](https://mlsanity.com/paper/2401.00391/) (2023)
- [Scenario Dreamer: Vectorized Latent Diffusion for Generating Driving Simulation Environments](https://mlsanity.com/paper/2503.22496/) (2025)
- [Building reliable sim driving agents by scaling self-play](https://mlsanity.com/paper/2502.14706/) (2025)
- [Promptable Closed-loop Traffic Simulation](https://mlsanity.com/paper/2409.05863/) (2024)
- [KiGRAS: Kinematic-Driven Generative Model for Realistic Agent Simulation](https://mlsanity.com/paper/2407.12940/) (2024)
- [Trajectory Prediction Meets Large Language Models: A Survey](https://mlsanity.com/paper/2506.03408/) (2025)
- [Advancing Multi-agent Traffic Simulation via R1-Style Reinforcement Fine-Tuning](https://mlsanity.com/paper/2509.23993/) (2025)
- [UniMM: A Unified Mixture Model Framework for Multi-Agent Simulation](https://mlsanity.com/paper/2501.17015/) (2025)
- [Generating Multimodal Driving Scenes via Next-Scene Prediction](https://mlsanity.com/paper/2503.14945/) (2025)
- [Decoupled Diffusion Sparks Adaptive Scene Generation](https://mlsanity.com/paper/2504.10485/) (2025)
- [LANGTRAJ: Diffusion Model and Dataset for Language-Conditioned Trajectory Simulation](https://mlsanity.com/paper/2504.11521/) (2025)
- [Enhancing Autonomous Driving Safety with Collision Scenario Integration](https://mlsanity.com/paper/2503.03957/) (2025)
- [TrajTok: Technical Report for 2025 Waymo Open Sim Agents Challenge](https://mlsanity.com/paper/2506.21618/) (2025)
- [ModeSeq: Taming Sparse Multimodal Motion Prediction with Sequential Mode Modeling](https://mlsanity.com/paper/2411.11911/) (2024)
- [Continuous Locomotive Crowd Behavior Generation](https://mlsanity.com/paper/2504.04756/) (2025)
- [DONUT: A Decoder-Only Model for Trajectory Prediction](https://mlsanity.com/paper/2506.06854/) (2025)
- [When Planners Meet Reality: How Learned, Reactive Traffic Agents Shift nuPlan Benchmarks](https://mlsanity.com/paper/2510.14677/) (2025)
- [SPACeR: Self-Play Anchoring with Centralized Reference Models](https://mlsanity.com/paper/2510.18060/) (2025)
- [Learning to Drive in New Cities Without Human Demonstrations](https://mlsanity.com/paper/2602.15891/) (2026)
- [Direct Post-Training Preference Alignment for Multi-Agent Motion Generation Models Using Implicit Feedback from Pre-training Demonstrations](https://mlsanity.com/paper/2503.20105/) (2025)
- [DecompGAIL: Learning Realistic Traffic Behaviors with Decomposed Multi-Agent Generative Adversarial Imitation Learning](https://mlsanity.com/paper/2510.06913/) (2025)
- [Learning Through Retrospection: Improving Trajectory Prediction for Automated Driving with Error Feedback](https://mlsanity.com/paper/2504.13785/) (2025)
- [RIFT: Group-Relative RL Fine-Tuning for Realistic and Controllable Traffic Simulation](https://mlsanity.com/paper/2505.03344/) (2025)
- [Do LLM Modules Generalize? A Study on Motion Generation for Autonomous Driving](https://mlsanity.com/paper/2509.02754/) (2025)
- [Perfect Prediction or Plenty of Proposals? What Matters Most in Planning for Autonomous Driving](https://mlsanity.com/paper/2510.15505/) (2025)
- [nuPlan-R: A Closed-Loop Planning Benchmark for Autonomous Driving via Reactive Multi-Agent Simulation](https://mlsanity.com/paper/2511.10403/) (2025)

## Abstract (arXiv)

A longstanding challenge for self-driving development is simulating dynamic driving scenarios seeded from recorded driving logs. In pursuit of this functionality, we apply tools from discrete sequence modeling to model how vehicles, pedestrians and cyclists interact in driving scenarios. Using a simple data-driven tokenization scheme, we discretize trajectories to centimeter-level resolution using a small vocabulary. We then model the multi-agent sequence of discrete motion tokens with a GPT-like encoder-decoder that is autoregressive in time and takes into account intra-timestep interaction between agents. Scenarios sampled from our model exhibit state-of-the-art realism; our model tops the Waymo Sim Agents Benchmark, surpassing prior work along the realism meta metric by 3.3% and along the interaction metric by 9.9%. We ablate our modeling choices in full autonomy and partial autonomy settings, and show that the representations learned by our model can quickly be adapted to improve performance on nuScenes. We additionally evaluate the scalability of our model with respect to parameter count and dataset size, and use density estimates from our model to quantify the saliency of context length and intra-timestep interaction for the traffic modeling task.