# Multiverse Transformer: 1st Place Solution for Waymo Open Sim Agents Challenge 2023

arXiv: https://arxiv.org/abs/2306.11868 · HTML: https://mlsanity.com/paper/2306.11868/

## Key facts

- Cited by 25 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 20 references that are in the map.
- First posted on arXiv 20 Jun 2023.

## Builds on (most cited first, 20 of 20)

- [Large Scale Interactive Motion Forecasting for Autonomous Driving : The Waymo Open Motion Dataset](https://mlsanity.com/paper/2104.10133/) (2021)
- [VectorNet: Encoding HD Maps and Agent Dynamics from Vectorized Representation](https://mlsanity.com/paper/2005.04259/) (2020)
- [AgentFormer: Agent-Aware Transformers for Socio-Temporal Multi-Agent Forecasting](https://mlsanity.com/paper/2103.14023/) (2021)
- [Wayformer: Motion Forecasting via Simple & Efficient Attention Networks](https://mlsanity.com/paper/2207.05844/) (2022)
- [Motion Transformer with Global Intention Localization and Local Movement Refinement](https://mlsanity.com/paper/2209.13508/) (2022)
- [MotionDiffuser: Controllable Multi-Agent Motion Prediction using Diffusion](https://mlsanity.com/paper/2306.03083/) (2023)
- [TrafficSim: Learning to Simulate Realistic Multi-Agent Behaviors](https://mlsanity.com/paper/2101.06557/) (2021)
- [Guided Conditional Diffusion for Controllable Traffic Simulation](https://mlsanity.com/paper/2210.17366/) (2022)
- [Scene Transformer: A unified architecture for predicting multiple agent trajectories](https://mlsanity.com/paper/2106.08417/) (2021)
- [Generating Useful Accident-Prone Driving Scenarios via a Learned Traffic Prior](https://mlsanity.com/paper/2112.05077/) (2021)
- [Trace and Pace: Controllable Pedestrian Animation via Guided Trajectory Diffusion](https://mlsanity.com/paper/2304.01893/) (2023)
- [Latent Variable Sequential Set Transformers For Joint Multi-Agent Motion Prediction](https://mlsanity.com/paper/2104.00563/) (2021)
- [Implicit Latent Variable Model for Scene-Consistent Motion Forecasting](https://mlsanity.com/paper/2007.12036/) (2020)
- [TrafficGen: Learning to Generate Diverse and Realistic Traffic Scenarios](https://mlsanity.com/paper/2210.06609/) (2022)
- [The Waymo Open Sim Agents Challenge](https://mlsanity.com/paper/2305.12032/) (2023)
- [BITS: Bi-level Imitation for Traffic Simulation](https://mlsanity.com/paper/2208.12403/) (2022)
- [SimNet: Learning Reactive Self-driving Simulations from Real-world Observations](https://mlsanity.com/paper/2105.12332/) (2021)
- [Symphony: Learning Realistic and Diverse Agents for Autonomous Driving Simulation](https://mlsanity.com/paper/2205.03195/) (2022)
- [JFP: Joint Future Prediction with Interactive Multi-Agent Modeling for Autonomous Driving](https://mlsanity.com/paper/2212.08710/) (2022)
- [PreTR: Spatio-Temporal Non-Autoregressive Trajectory Prediction Transformer](https://mlsanity.com/paper/2203.09293/) (2022)

## Built on by (most cited first, 25 of 25)

- [Robust Autonomy Emerges from Self-Play](https://mlsanity.com/paper/2502.03349/) (2025)
- [BehaviorGPT: Smart Agent Simulation for Autonomous Driving with Next-Patch Prediction](https://mlsanity.com/paper/2405.17372/) (2024)
- [SceneDiffuser: Efficient and Controllable Driving Simulation Initialization and Rollout](https://mlsanity.com/paper/2412.12129/) (2024)
- [Solving Motion Planning Tasks with a Scalable Generative Model](https://mlsanity.com/paper/2407.02797/) (2024)
- [WcDT: World-centric Diffusion Transformer for Traffic Scene Generation](https://mlsanity.com/paper/2404.02082/) (2024)
- [Scenario Dreamer: Vectorized Latent Diffusion for Generating Driving Simulation Environments](https://mlsanity.com/paper/2503.22496/) (2025)
- [CtRL-Sim: Reactive and Controllable Driving Agents with Offline Reinforcement Learning](https://mlsanity.com/paper/2403.19918/) (2024)
- [Improving Agent Behaviors with RL Fine-tuning for Autonomous Driving](https://mlsanity.com/paper/2409.18343/) (2024)
- [SMART: Scalable Multi-agent Real-time Motion Generation via Next-token Prediction](https://mlsanity.com/paper/2405.15677/) (2024)
- [KiGRAS: Kinematic-Driven Generative Model for Realistic Agent Simulation](https://mlsanity.com/paper/2407.12940/) (2024)
- [TrafficBots V1.5: Traffic Simulation via Conditional VAEs and Transformers with Relative Pose Encoding](https://mlsanity.com/paper/2406.10898/) (2024)
- [UniMM: A Unified Mixture Model Framework for Multi-Agent Simulation](https://mlsanity.com/paper/2501.17015/) (2025)
- [DRoPE: Directional Rotary Position Embedding for Efficient Agent Interaction Modeling](https://mlsanity.com/paper/2503.15029/) (2025)
- [When Planners Meet Reality: How Learned, Reactive Traffic Agents Shift nuPlan Benchmarks](https://mlsanity.com/paper/2510.14677/) (2025)
- [SceneStreamer: Continuous Scenario Generation as Next Token Group Prediction](https://mlsanity.com/paper/2506.23316/) (2025)
- [Post-Training and Test-Time Scaling of Generative Agent Behavior Models for Interactive Autonomous Driving](https://mlsanity.com/paper/2512.13262/) (2025)
- [TSDiT: Traffic Scene Diffusion Models With Transformers](https://mlsanity.com/paper/2405.02289/) (2023)
- [Model Predictive Simulation Using Structured Graphical Models and Transformers](https://mlsanity.com/paper/2406.19635/) (2024)
- [Autonomous Vehicle Controllers From End-to-End Differentiable Simulation](https://mlsanity.com/paper/2409.07965/) (2024)
- [Unlocking Efficient Vehicle Dynamics Modeling via Analytic World Models](https://mlsanity.com/paper/2502.10012/) (2025)
- [Controllable risk scenario generation from human crash data for autonomous vehicle testing](https://mlsanity.com/paper/2512.07874/) (2025)
- [Long-term Traffic Simulation via Structured Autoregressive Modeling](https://mlsanity.com/paper/2606.31209/) (2026)
- [Bench4Merge: A Comprehensive Benchmark for Merging in Realistic Dense Traffic with Micro-Interactive Vehicles](https://mlsanity.com/paper/2410.15912/) (2024)
- [Benchmarking Autonomous Driving Planners Across Leaderboards: A Unified CARLA-Based Evaluation](https://mlsanity.com/paper/2509.22754/) (2025)
- [Bridging Local Observation and Global Simulation in Closed-Loop Traffic Modeling](https://mlsanity.com/paper/2606.31844/) (2026)

## Abstract (arXiv)

This technical report presents our 1st place solution for the Waymo Open Sim Agents Challenge (WOSAC) 2023. Our proposed MultiVerse Transformer for Agent simulation (MVTA) effectively leverages transformer-based motion prediction approaches, and is tailored for closed-loop simulation of agents. In order to produce simulations with a high degree of realism, we design novel training and sampling methods, and implement a receding horizon prediction mechanism. In addition, we introduce a variable-length history aggregation method to mitigate the compounding error that can arise during closed-loop autoregressive execution. On the WOSAC, our MVTA and its enhanced version MVTE reach a realism meta-metric of 0.5091 and 0.5168, respectively, outperforming all the other methods on the leaderboard.