mlsanityData to 7 Oct 2026

PhysGen: Rigid-Body Physics-Grounded Image-to-Video Generation

arXiv 2409.18964 ↗First posted 27 Sep 2024cs.CV cs.AI cs.LGcited by 169

Not on the map. Why?

Cited by 39 in Diffusion models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We present PhysGen, a novel image-to-video generation method that converts a single image and an input condition (e.g., force and torque applied to an object in the image) to produce a realistic, physically plausible, and temporally consistent video. Our key insight is to integrate model-based physical simulation with a data-driven video generation process, enabling plausible image-space dynamics. At the heart of our system are three core components: (i) an image understanding module that effectively captures the geometry, materials, and physical parameters of the image; (ii) an image-space dynamics simulation model that utilizes rigid-body physics and inferred parameters to simulate realistic behaviors; and (iii) an image-based rendering and refinement module that leverages generative video diffusion to produce realistic video footage featuring the simulated motion. The resulting videos are realistic in both physics and appearance and are even precisely controllable, showcasing superior results over existing data-driven image-to-video generation works through quantitative comparison and comprehensive user study. PhysGen's resulting videos can be used for various downstream applications, such as turning an image into a realistic animation or allowing users to interact with the image and create various dynamics. Project page: https://stevenlsw.github.io/physgen/

Key facts. Cited by 169 papers in the map (data through 7 Oct 2026). Builds on 69 of its 94 references that are in the map. First posted on arXiv 27 Sep 2024.

Builds on

69 of 94 references.

References · 69

  1. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  2. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  3. 2023GPT-4 Technical Reportcited by 18,132
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  5. 2020Denoising Diffusion Implicit Modelscited by 9,655
  6. 2023Segment Anythingcited by 8,158
  7. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  8. 2022Scalable Diffusion Models with Transformerscited by 5,931
  9. 2019Generative Modeling by Estimating Gradients of the Data Distributioncited by 4,067
  10. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,274
  11. 2023Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasetscited by 2,526
  12. 2020RAFT: Recurrent All-Pairs Field Transforms for Optical Flowcited by 2,385
38 more
  1. 2021SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equationscited by 1,903
  2. 2022Imagen Video: High Definition Video Generation with Diffusion Modelscited by 1,765
  3. 2022Make-A-Video: Text-to-Video Generation without Text-Video Datacited by 1,725
  4. 2023Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Modelscited by 1,465
  5. 2023AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuningcited by 1,418
  6. 2018Towards Accurate Generative Models of Video: A New Metric & Challengescited by 1,204
  7. 2024Grounded SAM: Assembling Open-World Models for Diverse Visual Taskscited by 914
  8. 2023Language Model Beats Diffusion -- Tokenizer is Key to Visual Generationcited by 633
  9. 2023VideoCrafter1: Open Diffusion Models for High-Quality Video Generationcited by 530
  10. 2016Interaction Networks for Learning about Objects, Relations and Physicscited by 503
  11. 2023VideoPoet: A Large Language Model for Zero-Shot Video Generationcited by 479
  12. 2022Phenaki: Variable Length Video Generation From Open Domain Textual Descriptioncited by 475
  13. 2023DynamiCrafter: Animating Open-domain Images with Video Diffusion Priorscited by 463
  14. 2023VideoComposer: Compositional Video Synthesis with Motion Controllabilitycited by 461
  15. 2024Lumiere: A Space-Time Diffusion Model for Video Generationcited by 450
  16. 2023PhysGaussian: Physics-Integrated 3D Gaussians for Generative Dynamicscited by 368
  17. 2023LAVIE: High-Quality Video Generation with Cascaded Latent Diffusion Modelscited by 349
  18. 2023I2VGen-XL: High-Quality Image-to-Video Synthesis via Cascaded Diffusion Modelscited by 343
  19. 2021Omnidata: A Scalable Pipeline for Making Multi-Task Mid-Level Vision Datasets from 3D Scanscited by 336
  20. 2023Photorealistic Video Generation with Diffusion Modelscited by 304
  21. 2023Set-of-Mark Prompting Unleashes Extraordinary Visual Grounding in GPT-4Vcited by 291
  22. 2023Preserve Your Own Correlation: A Noise Prior for Video Diffusion Modelscited by 284
  23. 2024GeoWizard: Unleashing the Diffusion Priors for 3D Geometry Estimation from a Single Imagecited by 283
  24. 2023Emu Video: Factorizing Text-to-Video Generation by Explicit Image Conditioningcited by 254
  25. 2019DiffTaichi: Differentiable Programming for Physical Simulationcited by 248
  26. 2018Graph networks as learnable physics engines for inference and controlcited by 242
  27. 2023Inpaint Anything: Segment Anything Meets Image Inpaintingcited by 233
  28. 2023SEINE: Short-to-Long Video Diffusion Model for Generative Transition and Predictioncited by 231
  29. 2018Learning Particle Dynamics for Manipulating Rigid Bodies, Deformable Objects, and Fluidscited by 219
  30. 2022Thin-Plate Spline Motion Model for Image Animationcited by 209
  31. 2023DreamVideo: Composing Your Dream Videos with Customized Subject and Motioncited by 186
  32. 2023PAC-NeRF: Physics Augmented Continuum Neural Radiance Fields for Geometry-Agnostic System Identificationcited by 145
  33. 2020Unpaired Motion Style Transfer from Video to Animationcited by 94
  34. 2023Generative Image Dynamicscited by 93
  35. 2018Flexible Neural Representation for Physics Predictioncited by 91
  36. 2019DensePhysNet: Learning Dense Physical Object Representations via Multi-step Dynamic Interactionscited by 61
  37. 2023GPT4Motion: Scripting Physical Motions in Text-to-Video Generation via Blender-Oriented GPT Planningcited by 61
  38. 2018Photo Wake-Up: 3D Character Animation from a Single Photocited by 60

50 most cited of 69 All 69 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 15; 2025: 74; 2026: 80.

169 papers.

Sort:

By area:

  1. 2024Motion Prompting: Controlling Video Generation with Motion Trajectoriescited by 176
  2. 2024Towards World Simulator: Crafting Physical Commonsense-Based Benchmark for Video Generationcited by 167
  3. 2024Understanding World or Predicting Future? A Comprehensive Survey of World Modelscited by 161
  4. 2025VideoPhy-2: A Challenging Action-Centric Physical Commonsense Evaluation in Video Generationcited by 141
  5. 2025VideoJAM: Joint Appearance-Motion Representations for Enhanced Motion Generation in Video Modelscited by 112
  6. 2025VideoREPA: Learning Physics for Video Generation through Relational Alignment with Foundation Modelscited by 88
  7. 2025WISA: World Simulator Assistant for Physics-Aware Text-to-Video Generationcited by 87
  8. 2024PhyT2V: LLM-Guided Iterative Self-Refinement for Physics-Grounded Text-to-Video Generationcited by 77
  9. 2025PhysGen3D: Crafting a Miniature Interactive World from a Single Imagecited by 65
  10. 2025Force Prompting: Video Generation Models Can Learn and Generalize Physics-based Control Signalscited by 51
  11. 2025VLIPP: Towards Physically Plausible Video Generation with Vision and Language Informed Physical Priorcited by 49
  12. 2024PhysMotion: Physics-Grounded Dynamics From a Single Imagecited by 45
38 more
  1. 2024Improving Dynamic Object Interactions in Text-to-Video Generation with AI Feedbackcited by 45
  2. 2025WonderPlay: Dynamic 3D Scene Generation from a Single Image and Actionscited by 43
  3. 2025PhysAnimator: Physics-Guided Generative Cartoon Animationcited by 40
  4. 2025Inference-Time Text-to-Video Alignment with Diffusion Latent Beam Searchcited by 40
  5. 2025NewtonGen: Physics-Consistent and Controllable Text-to-Video Generation via Neural Newtonian Dynamicscited by 40
  6. 2025PhysCtrl: Generative Physics for Controllable and Physics-Grounded Video Generationcited by 39
  7. 2025RoboScape: Physics-informed Embodied World Modelcited by 38
  8. 2024PhysFlow: Unleashing the Potential of Multi-modal Foundation Models and Video Diffusion for 4D Dynamic Physical Scene Simulationcited by 36
  9. 2025Exploring the Evolution of Physics Cognition in Video Generation: A Surveycited by 31
  10. 2024AutoVFX: Physically Realistic Video Editing from Natural Language Instructionscited by 27
  11. 2025Real-to-Sim Robot Policy Evaluation with Gaussian Splatting Simulation of Soft-Body Interactionscited by 27
  12. 2025Generative Physical AI in Vision: A Surveycited by 24
  13. 2024Phys4DGen: Physics-Compliant 4D Generation with Multi-Material Composition Perceptioncited by 23
  14. 2025A Survey of Interactive Generative Videocited by 23
  15. 2025What about gravity in video generation? Post-Training Newton's Laws with Verifiable Rewardscited by 23
  16. 2025FlowMo: Variance-Based Flow Guidance for Coherent Motion in Video Generationcited by 21
  17. 2025Hierarchical Fine-grained Preference Optimization for Physically Plausible Video Generationcited by 20
  18. 2025Sparse identification of nonlinear dynamics and Koopman operators with Shallow Recurrent Decoder Networkscited by 19
  19. 2024A Survey of Sustainability in Large Language Models: Applications, Economics, and Challengescited by 18
  20. 2025Vid2Sim: Generalizable, Video-based Reconstruction of Appearance, Geometry and Physics for Mesh-free Simulationcited by 18
  21. 2025TiViBench: Benchmarking Think-in-Video Reasoning for Video Generative Modelscited by 18
  22. 2026RealWonder: Real-Time Physical Action-Conditioned Video Generationcited by 18
  23. 2025I2-NeRF: Learning Neural Radiance Fields Under Physically-Grounded Media Interactionscited by 17
  24. 2025DecoupledGaussian: Object-Scene Decoupling for Physics-Based Interactioncited by 16
  25. 2026PlayWorld: Learning Robot World Models from Autonomous Playcited by 16
  26. 2025Synthetic Video Enhances Physical Fidelity in Video Synthesiscited by 15
  27. 2025Simulating the Visual World with Artificial Intelligence: A Roadmapcited by 15
  28. 2026PhysRVG: Physics-Aware Unified Reinforcement Learning for Video Generative Modelscited by 15
  29. 2025PAT3D: Physics-Augmented Text-to-3D Scene Generationcited by 14
  30. 2026Self-Refining Video Samplingcited by 14
  31. 2025Position: Interactive Generative Video as Next-Generation Game Enginecited by 13
  32. 2025FreeGave: 3D Physics Learning from Dynamic Videos by Gaussian Velocitycited by 13
  33. 2025Enhancing Physical Plausibility in Video Generation by Reasoning the Implausibilitycited by 13
  34. 2025HoloScene: Simulation-Ready Interactive 3D Worlds from a Single Videocited by 13
  35. 2025MoAlign: Motion-Centric Representation Alignment for Video Diffusion Modelscited by 13
  36. 2026ActWorld: From Explorable to Interactive World Model via Action-Aware Memorycited by 13
  37. 2024MiraGe: Editable 2D Images using Gaussian Splattingcited by 12
  38. 2025RULER-Bench: Probing Rule-based Reasoning Abilities of Next-level Video Generation Models for Vision Foundation Intelligencecited by 12

50 most cited of 169 All 169 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026PhyCo: Learning Controllable Physical Priors for Generative Motion
  2. 2025"PhyWorldBench": A Comprehensive Evaluation of Physical Realism in Text-to-Video Models
  3. 2026RigidBench: Evaluating Rigid-Body Physics in Video Generation Models
  4. 2025PhysID: Physics-based Interactive Dynamics from a Single-view Image
  5. 2026PhysRAG: Enhancing Physics-Awareness in Video Generation via Retrieval-Augmented Generation
  6. 2026PhyGround: Benchmarking Physical Reasoning in Generative World Models
  7. 2025Evaluating Newtonian Mechanics in Video Generative Models with Real Physical Systems
  8. 2024PhysDreamer: Physics-Based Interaction with 3D Objects via Video Generation