mlsanityData to 7 Oct 2026

WorldSimBench: Towards Video Generation Models as World Simulators

arXiv 2410.18072 ↗First posted 23 Oct 2024cs.CVcited by 867

On the map: World models → Video-generation world models and simulators

Abstract (arXiv)

Recent advancements in predictive models have demonstrated exceptional capabilities in predicting the future state of objects and scenes. However, the lack of categorization based on inherent characteristics continues to hinder the progress of predictive model development. Additionally, existing benchmarks are unable to effectively evaluate higher-capability, highly embodied predictive models from an embodied perspective. In this work, we classify the functionalities of predictive models into a hierarchy and take the first step in evaluating World Simulators by proposing a dual evaluation framework called WorldSimBench. WorldSimBench includes Explicit Perceptual Evaluation and Implicit Manipulative Evaluation, encompassing human preference assessments from the visual perspective and action-level evaluations in embodied tasks, covering three representative embodied scenarios: Open-Ended Embodied Environment, Autonomous, Driving, and Robot Manipulation. In the Explicit Perceptual Evaluation, we introduce the HF-Embodied Dataset, a video assessment dataset based on fine-grained human feedback, which we use to train a Human Preference Evaluator that aligns with human perception and explicitly assesses the visual fidelity of World Simulators. In the Implicit Manipulative Evaluation, we assess the video-action consistency of World Simulators by evaluating whether the generated situation-aware video can be accurately translated into the correct control signals in dynamic environments. Our comprehensive evaluation offers key insights that can drive further innovation in video generation models, positioning World Simulators as a pivotal advancement toward embodied artificial intelligence.

Key facts. Cited by 867 papers in the map (data through 7 Oct 2026). Placed in Video-generation world models and simulators, one of 11 directions in World models. Builds on 46 of its 52 references that are in the map. First posted on arXiv 23 Oct 2024.

Builds on

46 of 52 references.

Closest ancestors · 2

Same direction, newest first

  1. 2023Learning Interactive Real-World Simulators9 Oct
  2. 2022Scalable Diffusion Models with Transformers19 Dec

Other references · 44

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2023Visual Instruction Tuningcited by 7,792
  5. 2019nuScenes: A multimodal dataset for autonomous drivingcited by 4,485
  6. 2022InstructPix2Pix: Learning to Follow Image Editing Instructionscited by 2,591
  7. 2017CARLA: An Open Urban Driving Simulatorcited by 2,431
  8. 2023Voyager: An Open-Ended Embodied Agent with Large Language Modelscited by 2,081
  9. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,037
  10. 2023VBench: Comprehensive Benchmark Suite for Video Generative Modelscited by 1,755
  11. 2021Ego4D: Around the World in 3,000 Hours of Egocentric Videocited by 1,530
  12. 2018World Modelscited by 1,444
32 more
  1. 2023AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuningcited by 1,418
  2. 2023AgentBench: Evaluating LLMs as Agentscited by 1,211
  3. 2021Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrievalcited by 1,175
  4. 2017The "something something" video database for learning and evaluating visual common sensecited by 1,092
  5. 2022Planning with Diffusion for Flexible Behavior Synthesiscited by 1,023
  6. 2023ModelScope Text-to-Video Technical Reportcited by 651
  7. 2021CALVIN: A Benchmark for Language-Conditioned Policy Learning for Long-Horizon Robot Manipulation Taskscited by 564
  8. 2023Learning Universal Policies via Text-Guided Video Generationcited by 506
  9. 2023DynamiCrafter: Animating Open-domain Images with Video Diffusion Priorscited by 463
  10. 2023Describe, Explain, Plan and Select: Interactive Planning with Large Language Models Enables Open-World Multi-Task Agentscited by 396
  11. 2023LAVIE: High-Quality Video Generation with Cascaded Latent Diffusion Modelscited by 349
  12. 2022Video PreTraining (VPT): Learning to Act by Watching Unlabeled Online Videoscited by 338
  13. 2024Vista: A Generalizable Driving World Model with High Fidelity and Versatile Controllabilitycited by 337
  14. 2023EvalCrafter: Benchmarking and Evaluating Large Video Generation Modelscited by 332
  15. 2023LMDrive: Closed-Loop End-to-End Driving with Large Language Modelscited by 301
  16. 2023Zero-Shot Robotic Manipulation with Pretrained Image-Editing Diffusion Modelscited by 264
  17. 2021Bridge Data: Boosting Generalization of Robotic Skills with Cross-Domain Datasetscited by 248
  18. 2019MineRL: A Large-Scale Dataset of Minecraft Demonstrationscited by 210
  19. 2023LAMM: Language-Assisted Multi-Modal Instruction-Tuning Dataset, Framework, and Benchmarkcited by 169
  20. 2023Video Language Planningcited by 160
  21. 2024Flash-VStream: Memory-Based Real-Time Understanding for Long Video Streamscited by 160
  22. 2023ManipLLM: Embodied Multimodal Large Language Model for Object-Centric Robotic Manipulationcited by 157
  23. 2024VisualAgentBench: Towards Large Multimodal Models as Visual Foundation Agentscited by 101
  24. 2023STEVE-1: A Generative Model for Text-to-Behavior in Minecraftcited by 95
  25. 2024EasyAnimate: High-Performance Video Generation Framework with Hybrid Windows Attention and Reward Backpropagationcited by 76
  26. 2023MP5: A Multi-modal Open-ended Embodied System in Minecraft via Active Perceptioncited by 67
  27. 2024MineDreamer: Learning to Follow Instructions via Chain-of-Imagination for Simulated-World Controlcited by 52
  28. 2024MMWorld: Towards Multi-discipline Multi-faceted World Model Evaluation in Videoscited by 43
  29. 2024Assessment of Multimodal Large Language Models in Alignment with Human Valuescited by 36
  30. 2023LEGO: Learning EGOcentric Action Frame Generation via Visual Instruction Tuningcited by 28
  31. 2024RH20T-P: A Primitive-Level Robotic Dataset Towards Composable Generalization Agentscited by 20
  32. 2024AD-H: Language-guided Autonomous Driving with Hierarchical Agentscited by 15

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 73; 2025: 414; 2026: 380.

867 papers.

Sort:

By area:

  1. 2025GR00T N1: An Open Foundation Model for Generalist Humanoid Robotscited by 1,167
  2. 2025Large Language Diffusion Modelscited by 872
  3. 2024Infinity: Scaling Bitwise AutoRegressive Modeling for High-Resolution Image Synthesiscited by 297
  4. 2026Causal World Modeling for Robot Controlcited by 289
  5. 2024Open-Sora Plan: Open-Source Large Video Generation Modelcited by 286
  6. 2025Improving Video Generation with Human Feedbackcited by 267
  7. 2025Mercury: Ultra-Fast Language Models Based on Diffusioncited by 225
  8. 2024CBraMod: A Criss-Cross Brain Foundation Model for EEG Decodingcited by 188
  9. 2024Motion Prompting: Controlling Video Generation with Motion Trajectoriescited by 176
  10. 2025Self-Forcing++: Towards Minute-Scale High-Quality Video Generationcited by 176
  11. 2025History-Guided Video Diffusioncited by 165
  12. 2024Understanding World or Predicting Future? A Comprehensive Survey of World Modelscited by 161
38 more
  1. 2025Open-Sora 2.0: Training a Commercial-Level Video Generation Model in 200kcited by 157
  2. 2025Context as Memory: Scene-Consistent Interactive Long Video Generation with Memory Retrievalcited by 157
  3. 2024Identity-Preserving Text-to-Video Generation by Frequency Decompositioncited by 155
  4. 2024AC3D: Analyzing and Improving 3D Camera Control in Video Diffusion Transformerscited by 151
  5. 2026Advancing Open-source World Modelscited by 144
  6. 2025Diffusion Adversarial Post-Training for One-Step Video Generationcited by 142
  7. 2025VideoPhy-2: A Challenging Action-Centric Physical Commonsense Evaluation in Video Generationcited by 141
  8. 2025OmniHuman-1: Rethinking the Scaling-Up of One-Stage Conditioned Human Animation Modelscited by 127
  9. 2025RoboRefer: Towards Spatial Referring with Reasoning in Vision-Language Models for Roboticscited by 125
  10. 2025GameFactory: Creating New Games with Generative Interactive Videoscited by 124
  11. 2025Video World Models with Long-term Spatial Memorycited by 124
  12. 2025Test-Time Training Done Rightcited by 121
  13. 2025Long-Context Autoregressive Video Modeling with Next-Frame Predictioncited by 119
  14. 2025Go-with-the-Flow: Motion-Controllable Video Diffusion Models Using Real-Time Warped Noisecited by 114
  15. 2025VideoJAM: Joint Appearance-Motion Representations for Enhanced Motion Generation in Video Modelscited by 112
  16. 2025Inductive Moment Matchingcited by 107
  17. 2024Adaptive Caching for Faster Video Generation with Diffusion Transformerscited by 100
  18. 2024Wonderland: Navigating 3D Scenes from a Single Imagecited by 98
  19. 2025MineWorld: a Real-Time and Open-Source Interactive World Model on Minecraftcited by 93
  20. 2025Voyager: Long-Range and World-Consistent Video Diffusion for Explorable 3D Scene Generationcited by 93
  21. 2024SynCamMaster: Synchronizing Multi-Camera Video Generation from Diverse Viewpointscited by 92
  22. 2025SANA-Video: Efficient Video Generation with Block Linear Diffusion Transformercited by 91
  23. 2025TesserAct: Learning 4D Embodied World Modelscited by 90
  24. 2025Perception, Reason, Think, and Plan: A Survey on Large Multimodal Reasoning Modelscited by 89
  25. 2025Flow Matching Policy Gradientscited by 86
  26. 2025UniVideo: Unified Understanding, Generation, and Editing for Videoscited by 85
  27. 2025Autoregressive Adversarial Post-Training for Real-Time Interactive Video Generationcited by 84
  28. 2024One-Step Diffusion Policy: Fast Visuomotor Policies via Diffusion Distillationcited by 83
  29. 2025VideoPainter: Any-length Video Inpainting and Editing with Plug-and-Play Context Controlcited by 83
  30. 2025SONIC: Supersizing Motion Tracking for Natural Humanoid Whole-Body Controlcited by 83
  31. 2025Align Your Flow: Scaling Continuous-Time Flow Map Distillationcited by 81
  32. 2025Large Scale Diffusion Distillation via Score-Regularized Continuous-Time Consistencycited by 81
  33. 2025Wan-Animate: Unified Character Animation and Replacement with Holistic Replicationcited by 76
  34. 2025All-atom Diffusion Transformers: Unified generative modelling of molecules and materialscited by 75
  35. 2025A Survey on Diffusion Language Modelscited by 75
  36. 2025Reward Forcing: Efficient Streaming Video Generation with Rewarded Distribution Matching Distillationcited by 75
  37. 2025Improving the Diffusability of Autoencoderscited by 74
  38. 2024DiTCtrl: Exploring Attention Control in Multi-Modal Diffusion Transformer for Tuning-Free Multi-Prompt Longer Video Generationcited by 73

50 most cited of 867 All 867 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024EVA: An Embodied World Model for Future Video Anticipation
  2. 2025WorldModelBench: Judging Video Generation Models As World Models
  3. 2026Wow, wo, val! A Comprehensive Embodied World Model Evaluation Turing Test
  4. 2026Video2World: Benchmarking Coding Agents for Interactive World Modeling from Embodied Videos
  5. 2025Towards High-Consistency Embodied World Model with Multi-View Trajectory Videos
  6. 2026How Should World Models Be Evaluated for Embodied Decision-Making? A Decision-Making-Centric Position
  7. 2025WorldLens: Full-Spectrum Evaluations of Driving World Models in Real World
  8. 2026WorldMark: A Unified Benchmark Suite for Interactive Video World Models