mlsanityData to 7 Oct 2026

CAT4D: Create Anything in 4D with Multi-View Video Diffusion Models

arXiv 2411.18613 ↗First posted 27 Nov 2024cs.CVcited by 160

On the map: Diffusion models → 3D, point cloud and scene generation

Abstract (arXiv)

We present CAT4D, a method for creating 4D (dynamic 3D) scenes from monocular video. CAT4D leverages a multi-view video diffusion model trained on a diverse combination of datasets to enable novel view synthesis at any specified camera poses and timestamps. Combined with a novel sampling approach, this model can transform a single monocular video into a multi-view video, enabling robust 4D reconstruction via optimization of a deformable 3D Gaussian representation. We demonstrate competitive performance on novel view synthesis and dynamic scene reconstruction benchmarks, and highlight the creative capabilities for 4D scene generation from real or generated videos. See our project page for results and interactive demos: https://cat-4d.github.io/.

Key facts. Cited by 160 papers in the map (data through 7 Oct 2026). Placed in 3D, point cloud and scene generation, one of 14 directions in Diffusion models. Builds on 79 of its 95 references that are in the map. First posted on arXiv 27 Nov 2024.

Builds on

79 of 95 references.

Closest ancestors · 16

Same direction, newest first

  1. 2024DimensionX: Create Any 3D and 4D Scenes from a Single Image with Controllable Video Diffusion7 Nov
  2. 2024ViewCrafter: Taming Video Diffusion Models for High-fidelity Novel View Synthesis3 Sep
  3. 2024Controlling Space and Time with Diffusion Models10 Jul
  4. 20244Real: Towards Photorealistic 4D Scene Generation via Video Diffusion Models11 Jun
  5. 2024Diffusion4D: Fast Spatial-temporal Consistent 4D Generation via Video Diffusion Models26 May
  6. 2024CAT3D: Create Anything in 3D with Multi-View Diffusion Models16 May
  7. 2024STAG4D: Spatial-Temporal Anchored Generative 4D Gaussians22 Mar
  8. 20234DGen: Grounded 4D Content Generation with Spatial-temporal Consistency28 Dec
  9. 2023Align Your Gaussians: Text-to-4D with Dynamic 3D Gaussians and Composed Diffusion Models21 Dec
  10. 2023ReconFusion: 3D Reconstruction with Diffusion Priors5 Dec
  11. 2023A Unified Approach for Text- and Image-guided 4D Scene Generation28 Nov
  12. 2023Animate124: Animating One Image to 4D Dynamic Scene24 Nov
4 more
  1. 2023SyncDreamer: Generating Multiview-consistent Images from a Single-view Image7 Sep
  2. 2023MVDream: Multi-view Diffusion for 3D Generation31 Aug
  3. 2023Zero-1-to-3: Zero-shot One Image to 3D Object20 Mar
  4. 2022DreamFusion: Text-to-3D using 2D Diffusion29 Sep

Other references · 63

  1. 2017Attention Is All You Needcited by 59,932
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  3. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,029
  4. 2020Denoising Diffusion Implicit Modelscited by 9,655
  5. 20233D Gaussian Splatting for Real-Time Radiance Field Renderingcited by 6,141
  6. 2023Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasetscited by 2,526
  7. 2021SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equationscited by 1,903
  8. 2022Make-A-Video: Text-to-Video Generation without Text-Video Datacited by 1,725
  9. 2022Objaverse: A Universe of Annotated 3D Objectscited by 1,522
  10. 2020D-NeRF: Neural Radiance Fields for Dynamic Scenescited by 1,459
  11. 2023AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuningcited by 1,418
  12. 2024Depth Anything: Unleashing the Power of Large-Scale Unlabeled Datacited by 1,346
38 more
  1. 2020Nerfies: Deformable Neural Radiance Fieldscited by 1,194
  2. 20234D Gaussian Splatting for Real-Time Dynamic Scene Renderingcited by 1,144
  3. 2023LRM: Large Reconstruction Model for Single Image to 3Dcited by 834
  4. 2020Neural Scene Flow Fields for Space-Time View Synthesis of Dynamic Scenescited by 750
  5. 2023K-Planes: Explicit Radiance Fields in Space, Time, and Appearancecited by 742
  6. 2023Dynamic 3D Gaussians: Tracking by Persistent Dynamic View Synthesiscited by 731
  7. 2021Common Objects in 3D: Large-Scale Learning and Evaluation of Real-life 3D Category Reconstructioncited by 646
  8. 2023HexPlane: A Fast Representation for Dynamic Scenescited by 582
  9. 2023CoTracker: It is Better to Track Togethercited by 516
  10. 2023Ego-Exo4D: Understanding Skilled Human Activity from First- and Third-Person Perspectivescited by 477
  11. 2024Lumiere: A Space-Time Diffusion Model for Video Generationcited by 450
  12. 2024CameraCtrl: Enabling Camera Control for Text-to-Video Generationcited by 448
  13. 2024MonST3R: A Simple Approach for Estimating Geometry in the Presence of Motioncited by 431
  14. 2021Dynamic View Synthesis from Dynamic Monocular Videocited by 420
  15. 2024UniDepth: Universal Monocular Metric Depth Estimationcited by 375
  16. 2022Kubric: A scalable dataset generatorcited by 373
  17. 2022Assembly101: A Large-Scale Multi-View Video Dataset for Understanding Procedural Activitiescited by 284
  18. 2023TAPIR: Tracking Any Point with per-frame Initialization and temporal Refinementcited by 281
  19. 2023PointOdyssey: A Large-Scale Synthetic Dataset for Long-Term Point Trackingcited by 274
  20. 2024Shape of Motion: 4D Reconstruction from a Single Videocited by 247
  21. 2023Track Anything: Segment Anything Meets Videoscited by 230
  22. 2022DynIBaR: Neural Dynamic Image-Based Renderingcited by 224
  23. 2023MVImgNet: A Large-scale Dataset of Multi-view Imagescited by 219
  24. 2023Text-To-4D Dynamic Scene Generationcited by 204
  25. 2022Monocular Dynamic View Synthesis: A Reality Checkcited by 199
  26. 2023DreamGaussian4D: Generative 4D Gaussian Splattingcited by 191
  27. 20234D-fy: Text-to-4D Generation Using Hybrid Score Distillation Samplingcited by 184
  28. 2024MoSca: Dynamic Gaussian Fusion from Casual Videos via 4D Motion Scaffoldscited by 177
  29. 2023HyperReel: High-Fidelity 6-DoF Video with Ray-Conditioned Samplingcited by 173
  30. 2024Direct-a-Video: Customized Video Generation with User-Directed Camera Movement and Object Motioncited by 167
  31. 2020Novel View Synthesis of Dynamic Scenes with Globally Coherent Depths from a Monocular Cameracited by 166
  32. 2022Fourier PlenOctrees for Dynamic Radiance Field Rendering in Real-timecited by 164
  33. 2024VD3D: Taming Large Video Diffusion Transformers for 3D Camera Controlcited by 156
  34. 2024L4GM: Large 4D Gaussian Reconstruction Modelcited by 143
  35. 2023Infinite Photorealistic Worlds using Procedural Generationcited by 142
  36. 2023Consistent4D: Consistent 360deg Dynamic Object Generation from Monocular Videocited by 124
  37. 2023DynMF: Neural Motion Factorization for Real-time Dynamic View Synthesis with 3D Gaussian Splattingcited by 124
  38. 2019Learning the Depths of Moving People by Watching Frozen Peoplecited by 120

50 most cited of 63 All 63 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 6; 2025: 93; 2026: 61.

160 papers.

Sort:

By area:

  1. 2025Continuous 3D Perception Model with Persistent Statecited by 527
  2. 2025ReCamMaster: Camera-Controlled Generative Rendering from A Single Videocited by 278
  3. 2024MegaSaM: Accurate, Fast, and Robust Structure and Motion from Casual Dynamic Videoscited by 224
  4. 2025ViPE: Video Pose Engine for 3D Geometric Perceptioncited by 162
  5. 2025TrajectoryCrafter: Redirecting Camera Trajectory for Monocular Videos via Diffusion Modelscited by 120
  6. 2025Go-with-the-Flow: Motion-Controllable Video Diffusion Models Using Real-Time Warped Noisecited by 114
  7. 2025Aether: Geometric-Aware Unified World Modelingcited by 94
  8. 2025Cameras as Relative Positional Encodingcited by 94
  9. 2024Stereo4D: Learning How Things Move in 3D from Internet Stereo Videoscited by 89
  10. 2025MotionStream: Real-Time Video Generation with Interactive Motion Controlscited by 71
  11. 2025EnerVerse: Envisioning Embodied Future Space for Robotics Manipulationcited by 59
  12. 2025Bolt3D: Generating 3D Scenes in Secondscited by 57
38 more
  1. 20254DGT: Learning a 4D Gaussian Transformer Using Real-World Monocular Videoscited by 46
  2. 2025SV4D 2.0: Enhancing Spatio-Temporal Consistency in Multi-View Video Diffusion for High-Quality 4D Generationcited by 44
  3. 2025WonderPlay: Dynamic 3D Scene Generation from a Single Image and Actionscited by 43
  4. 2025VLM4D: Towards Spatiotemporal Awareness in Vision Language Modelscited by 43
  5. 2025PhysCtrl: Generative Physics for Controllable and Physics-Grounded Video Generationcited by 39
  6. 2025Lyra: Generative 3D Scene Reconstruction via Video Diffusion Model Self-Distillationcited by 37
  7. 2025Reangle-A-Video: 4D Video Generation as Video-to-Video Translationcited by 35
  8. 2025Free4D: Tuning-free 4D Scene Generation with Spatial-Temporal Consistencycited by 34
  9. 2025Any4D: Unified Feed-Forward Metric 4D Reconstructioncited by 32
  10. 20254DNeX: Feed-Forward 4D Generative Modeling Made Easycited by 31
  11. 2025Emergent Temporal Correspondences from Video Diffusion Transformerscited by 30
  12. 2025Generative Physical AI in Vision: A Surveycited by 24
  13. 20253D Scene Generation: A Surveycited by 24
  14. 2025Advances in Feed-Forward 3D Reconstruction and View Synthesis: A Surveycited by 23
  15. 2025Generative Video Motion Editing with 3D Point Trackscited by 21
  16. 2025EPiC: Efficient Video Camera Control Learning with Precise Anchor-Video Guidancecited by 20
  17. 2025UniLat3D: Geometry-Appearance Unified Latents for Single-Stage 3D Generationcited by 20
  18. 2025ShapeGen4D: Towards High Quality 4D Shape Generation from Videoscited by 20
  19. 2025Positional Encoding Fieldcited by 20
  20. 2025Macro-from-Micro Planning for High-Quality and Parallelized Autoregressive Long Video Generationcited by 19
  21. 2025Diffuman4D: 4D Consistent Human View Synthesis from Sparse-View Videos with Spatio-Temporal Diffusion Modelscited by 17
  22. 2025Skyfall-GS: Synthesizing Immersive 3D Urban Scenes from Satellite Imagerycited by 16
  23. 20244D Gaussian Splatting: Modeling Dynamic Scenes with Native 4D Primitivescited by 15
  24. 2025Zero4D: Training-Free 4D Video Generation From Single Video Using Off-the-Shelf Video Diffusioncited by 15
  25. 2026Motion 3-to-4: 3D Motion Reconstruction for 4D Synthesiscited by 15
  26. 2026ActionMesh: Animated 3D Mesh Generation with Temporal 3D Diffusioncited by 15
  27. 2025Simulating the Real World: A Unified Survey of Multimodal Generative Modelscited by 14
  28. 2025Survey of Video Diffusion Models: Foundations, Implementations, and Applicationscited by 14
  29. 2025SteerX: Creating Any Camera-Free 3D and 4D Scenes with Geometric Steeringcited by 13
  30. 20254D-LRM: Large Space-Time Reconstruction Model From and To Any View at Any Timecited by 13
  31. 2025Reconstruct, Inpaint, Test-Time Finetune: Dynamic Novel-view Synthesis from Monocular Videoscited by 13
  32. 2026AnchorWeave: World-Consistent Video Generation with Retrieved Local Spatial Memoriescited by 13
  33. 20254Real-Video-V2: Fused View-Time Attention and Feedforward Reconstruction for 4D Scene Generationcited by 12
  34. 2025BulletTime: Decoupled Control of Time and Camera Pose for Video Generationcited by 12
  35. 2026AnyView: Synthesizing Any Novel View in Dynamic Scenescited by 12
  36. 2025Vivid4D: Improving 4D Reconstruction from Monocular Video by Video Inpaintingcited by 11
  37. 2025See4D: Pose-Free 4D Generation via Auto-Regressive Video Inpaintingcited by 11
  38. 2025Light-X: Generative 4D Video Rendering with Camera and Illumination Controlcited by 11

50 most cited of 160 All 160 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Full-4D: Generating Full-Scope 4D Scenes from a Single-View Video
  2. 2024SV4D: Dynamic 3D Content Generation with Multi-Frame and Multi-View Consistency
  3. 20254DVD: Cascaded Dense-view Video Diffusion Model for High-quality 4D Content Generation
  4. 20244Diffusion: Multi-view Video Diffusion Model for 4D Generation
  5. 2025CoCo4D: Comprehensive and Complex 4D Scene Generation
  6. 2023Make-It-4D: Synthesizing a Consistent Long-Term Dynamic Scene Video from a Single Image
  7. 20264DAnyone: Create Anyone in 4D from a Casual Monocular Video
  8. 2026Flex4DHuman: Flexible Multi-view Video Diffusion for 4D Human Reconstruction