mlsanityData to 7 Oct 2026

Trajectory World Models for Heterogeneous Environments

arXiv 2502.01366 ↗First posted 3 Feb 2025cs.LGcited by 4

On the map: World models → Latent predictive world models

Abstract (arXiv)

Heterogeneity in sensors and actuators across environments poses a significant challenge to building large-scale pre-trained world models on top of this low-dimensional sensor information. In this work, we explore pre-training world models for heterogeneous environments by addressing key transfer barriers in both data diversity and model flexibility. We introduce UniTraj, a unified dataset comprising over one million trajectories from 80 environments, designed to scale data while preserving critical diversity. Additionally, we propose TrajWorld, a novel architecture capable of flexibly handling varying sensor and actuator information and capturing environment dynamics in-context. Pre-training TrajWorld on UniTraj yields substantial gains in transition prediction, achieves a new state-of-the-art for off-policy evaluation, and also delivers superior online performance of model predictive control. To the best of our knowledge, this work, for the first time, demonstrates the transfer benefits of world models across heterogeneous and complex control environments. Code and data are available at https://github.com/thuml/TrajWorld.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Placed in Latent predictive world models, one of 11 directions in World models. Builds on 66 of its 75 references that are in the map. First posted on arXiv 3 Feb 2025.

Builds on

66 of 75 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024DINO-WM: World Models on Pre-trained Visual Features enable Zero-shot Planning7 Nov

Other references · 65

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2017Proximal Policy Optimization Algorithmscited by 14,441
  4. 2023Segment Anythingcited by 8,158
  5. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,764
  6. 2017On Calibration of Modern Neural Networkscited by 4,860
  7. 2024OpenVLA: An Open-Source Vision-Language-Action Modelcited by 2,777
  8. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  9. 2016OpenAI Gymcited by 1,684
  10. 2021ViViT: A Video Vision Transformercited by 1,610
  11. 2019Dream to Control: Learning Behaviors by Latent Imaginationcited by 1,598
  12. 2024Octo: An Open-Source Generalist Robot Policycited by 1,427
38 more
  1. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,406
  2. 2019Mastering Atari, Go, Chess and Shogi by Planning with a Learned Modelcited by 1,327
  3. 2019Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learningcited by 1,168
  4. 2018DeepMind Control Suitecited by 935
  5. 2025Cosmos World Foundation Model Platform for Physical AIcited by 833
  6. 2022A Generalist Agentcited by 802
  7. 2018Recurrent World Models Facilitate Policy Evolutioncited by 761
  8. 2019When to Trust Your Model: Model-Based Policy Optimizationcited by 753
  9. 2018Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Modelscited by 748
  10. 2024Genie: Generative Interactive Environmentscited by 706
  11. 2021Offline Reinforcement Learning as One Big Sequence Modeling Problemcited by 636
  12. 2023TD-MPC2: Scalable, Robust World Models for Continuous Controlcited by 482
  13. 2021Implicit Behavioral Cloningcited by 442
  14. 2019Axial Attention in Multidimensional Transformerscited by 350
  15. 2015Doubly Robust Off-policy Value Evaluation for Reinforcement Learningcited by 337
  16. 2024Latent Action Pretraining from Videoscited by 302
  17. 2022Behavior Transformers: Cloning k modes with one stonecited by 299
  18. 2024GR-2: A Generative Video-Language-Action Model with Web-Scale Knowledge for Robot Manipulationcited by 290
  19. 2022Wayformer: Motion Forecasting via Simple & Efficient Attention Networkscited by 279
  20. 2019Batch Policy Learning under Constraintscited by 242
  21. 2018Visual Foresight: Model-Based Deep Reinforcement Learning for Vision-Based Robotic Controlcited by 220
  22. 2018Model-Ensemble Trust-Region Policy Optimizationcited by 214
  23. 2018Distributed Distributional Deterministic Policy Gradientscited by 195
  24. 2021Open-Ended Learning Leads to Generally Capable Agentscited by 177
  25. 2024Scaling Proprioceptive-Visual Learning with Heterogeneous Pre-trained Transformerscited by 145
  26. 2024Is Your LLM Secretly a World Model of the Internet? Model-Based Planning for Web Agentscited by 144
  27. 2021URLB: Unsupervised Reinforcement Learning Benchmarkcited by 138
  28. 2024Stop Regressing: Training Value Functions via Classification for Scalable Deep RLcited by 136
  29. 2024iVideoGPT: Interactive VideoGPTs are Scalable World Modelscited by 136
  30. 2024Web Agents with World Models: Learning and Leveraging Environment Dynamics in Web Navigationcited by 116
  31. 2020One Policy to Control Them All: Shared Modular Policies for Agent-Agnostic Controlcited by 114
  32. 2022Reinforcement Learning with Action-Free Pre-Training from Videoscited by 107
  33. 2022Don't Change the Algorithm, Change the Data: Exploratory Data for Offline Reinforcement Learningcited by 93
  34. 2020Offline Meta-Reinforcement Learning with Advantage Weightingcited by 92
  35. 2021Benchmarks for Deep Off-Policy Evaluationcited by 89
  36. 2020Off-Policy Evaluation via the Regularized Lagrangiancited by 86
  37. 2019V-MPO: On-Policy Maximum a Posteriori Policy Optimization for Discrete and Continuous Controlcited by 85
  38. 2020Sample Factory: Egocentric 3D Control from Pixels at 100000 FPS with Asynchronous Reinforcement Learningcited by 84

50 most cited of 65 All 65 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 3; 2026: 1.

4 papers.

  1. 2025Vid2World: Crafting Video Diffusion Models to Interactive World Modelscited by 67
  2. 2025A Comprehensive Survey on World Models for Embodied AIcited by 65
  3. 2025RLVR-World: Training World Models with Reinforcement Learningcited by 38
  4. 2026WestWorld: A Knowledge-Encoded Scalable Trajectory World Model for Diverse Robotic Systemscited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2026HEAT: Heterogeneous End-to-End Autonomous Driving via Trajectory-Guided World Models
  2. 2025SmallWorlds: Assessing Dynamics Understanding of World Models in Isolated Environments
  3. 2026DWM: Separating World Effects from Actions in Latent World Models
  4. 2025Trends in Motion Prediction Toward Deployable and Generalizable Autonomy: A Revisit and Perspectives
  5. 2025Time-Aware World Model for Adaptive Prediction and Control
  6. 2021Dream to Explore: Adaptive Simulations for Autonomous Systems
  7. 2025From Forecasting to Planning: Policy World Model for Collaborative State-Action Prediction
  8. 2024Enhanced Prediction of Multi-Agent Trajectories via Control Inference and State-Space Dynamics