mlsanityData to 7 Oct 2026

Understanding Physical Dynamics with Counterfactual World Modeling

arXiv 2312.06721 ↗First posted 11 Dec 2023cs.CVcited by 11

On the map: World models → Latent predictive world models

Abstract (arXiv)

The ability to understand physical dynamics is critical for agents to act in the world. Here, we use Counterfactual World Modeling (CWM) to extract vision structures for dynamics understanding. CWM uses a temporally-factored masking policy for masked prediction of video data without annotations. This policy enables highly effective "counterfactual prompting" of the predictor, allowing a spectrum of visual structures to be extracted from a single pre-trained predictor without finetuning on annotated datasets. We demonstrate that these structures are useful for physical dynamics understanding, allowing CWM to achieve the state-of-the-art performance on the Physion benchmark.

Key facts. Cited by 11 papers in the map (data through 7 Oct 2026). Placed in Latent predictive world models, one of 11 directions in World models. Builds on 69 of its 88 references that are in the map. First posted on arXiv 11 Dec 2023.

Builds on

69 of 88 references.

Closest ancestors · 2

Same direction, newest first

  1. 2023Unifying (Machine) Vision via Counterfactual World Modeling2 Jun
  2. 2018Learning Latent Dynamics for Planning from Pixels12 Nov

Other references · 67

  1. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  3. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  4. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,764
  5. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  6. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  7. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  8. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  9. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 4,993
  10. 2020RAFT: Recurrent All-Pairs Field Transforms for Optical Flowcited by 2,385
  11. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,360
  12. 2017The Kinetics Human Action Video Datasetcited by 1,894
38 more
  1. 2018Unsupervised Feature Learning via Non-Parametric Instance-level Discriminationcited by 1,838
  2. 2019Dream to Control: Learning Behaviors by Latent Imaginationcited by 1,598
  3. 2018Unsupervised Representation Learning by Predicting Image Rotationscited by 1,586
  4. 2023Zero-1-to-3: Zero-shot One Image to 3D Objectcited by 1,483
  5. 2017Accurate, Large Minibatch SGD: Training ImageNet in 1 Hourcited by 1,479
  6. 2016Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzlescited by 1,379
  7. 2022VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Trainingcited by 1,305
  8. 2016Colorful Image Colorizationcited by 1,113
  9. 2017The "something something" video database for learning and evaluating visual common sensecited by 1,092
  10. 2015Unsupervised Visual Representation Learning by Context Predictioncited by 1,062
  11. 2017The 2017 DAVIS Challenge on Video Object Segmentationcited by 942
  12. 2022R3M: A Universal Visual Representation for Robot Manipulationcited by 678
  13. 2023VideoMAE V2: Scaling Video Masked Autoencoders with Dual Maskingcited by 516
  14. 2016Interaction Networks for Learning about Objects, Relations and Physicscited by 503
  15. 2022Masked Autoencoders As Spatiotemporal Learnerscited by 428
  16. 2020Spatiotemporal Contrastive Video Representation Learningcited by 372
  17. 2022MAGVIT: Masked Generative Video Transformercited by 341
  18. 2022MCVD: Masked Conditional Video Diffusion for Prediction, Generation, and Interpolationcited by 307
  19. 2017Temporal Segment Networks for Action Recognition in Videoscited by 279
  20. 2016Deep Visual Foresight for Planning Robot Motioncited by 276
  21. 2023PointOdyssey: A Large-Scale Synthetic Dataset for Long-Term Point Trackingcited by 274
  22. 2016Unsupervised Learning for Physical Interaction through Video Predictioncited by 256
  23. 2020ThreeDWorld: A Platform for Interactive Multi-Modal Physical Simulationcited by 249
  24. 2021Bridge Data: Boosting Generalization of Robotic Skills with Cross-Domain Datasetscited by 248
  25. 2018Graph networks as learnable physics engines for inference and controlcited by 242
  26. 2023SEINE: Short-to-Long Video Diffusion Model for Generative Transition and Predictioncited by 231
  27. 2018Learning Particle Dynamics for Manipulating Rigid Bodies, Deformable Objects, and Fluidscited by 219
  28. 2023Cut and Learn for Unsupervised Object Detection and Instance Segmentationcited by 186
  29. 2023Spring: A High-Resolution High-Detail Dataset and Benchmark for Scene Flow, Optical Flow and Stereocited by 170
  30. 2016Learning to Poke by Poking: Experiential Learning of Intuitive Physicscited by 152
  31. 2022SlotFormer: Unsupervised Visual Dynamics Simulation with Object-Centric Modelscited by 146
  32. 2021Physion: Evaluating Physical Prediction from Vision in Humans and Machinescited by 145
  33. 2016Learning Features by Watching Objects Movecited by 138
  34. 2019PHYRE: A New Benchmark for Physical Reasoningcited by 137
  35. 2021TCLR: Temporal Contrastive Learning for Video Representationcited by 136
  36. 2022Diffusion Models for Video Prediction and Infillingcited by 134
  37. 2022MaskViT: Masked Visual Pre-Training for Video Predictioncited by 112
  38. 2022FreeSOLO: Learning to Segment Objects without Annotationscited by 102

50 most cited of 67 All 67 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 6; 2026: 5.

11 papers.

  1. 2025VideoREPA: Learning Physics for Video Generation through Relational Alignment with Foundation Modelscited by 88
  2. 2026Masked Visual Actions for Unified World Modelingcited by 7
  3. 2025Point Prompting: Counterfactual Tracking with Video Diffusion Modelscited by 5
  4. 2025Self-Supervised Learning of Motion Concepts by Optimizing Counterfactualscited by 4
  5. 2025Agentic World Modeling for 6G: Near-Real-Time Generative State-Space Reasoningcited by 4
  6. 2026LaMo: Self-Supervised Latent Motion Priors for Physical Realism in Video Generationcited by 4
  7. 2025Discovering and using Spelke segmentscited by 2
  8. 2026CausalVAE as a Plug-in for World Models: Towards Reliable Counterfactual Dynamicscited by 2
  9. 2026Envisioning the Future, One Step at a Timecited by 2
  10. 2025Back to the Feature: Explaining Video Classifiers with Video Counterfactual Explanationscited by 1
  11. 2026MultiEcho: An Experimental Science of Learned Worldscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026CRONOS: Benchmarking Counterfactual Physical Consistency in Video Models
  2. 2025Counterfactual World Models via Digital Twin-conditioned Video Diffusion
  3. 2026Low-Rank Dynamics-Effective Latent Carriers for Counterfactual Rollout in Learned World Models
  4. 2026Counterfactual Action Evaluation, Observation Bottlenecks, and Representation Geometry in Joint-Embedding Predictive World Models
  5. 2026Learning Counterfactual World Models for Embodied Reasoning under Partial Observability
  6. 2026StrucPhysVideo: Learning Physical Dynamics from Structured Captions and Robot Actions
  7. 2026Physical Object Understanding with a Physically Controllable World Model
  8. 2026LPA-CWM: A Learned Physical Adjudicator for Motion Reasoning with Counterfactual World Models