# Beyond Pixels: From Video Priors to 4D Worlds

arXiv: https://arxiv.org/abs/2608.10744 · HTML: https://mlsanity.com/paper/2608.10744/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 29 of its 38 references that are in the map.
- First posted on arXiv 11 Aug 2026.

## Builds on (most cited first, 29 of 29)

- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [Wan: Open and Advanced Large-Scale Video Generative Models](https://mlsanity.com/paper/2503.20314/) (2025)
- [Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasets](https://mlsanity.com/paper/2311.15127/) (2023)
- [CogVideoX: Text-to-Video Diffusion Models with An Expert Transformer](https://mlsanity.com/paper/2408.06072/) (2024)
- [MotionCtrl: A Unified and Flexible Motion Controller for Video Generation](https://mlsanity.com/paper/2312.03641/) (2023)
- [CameraCtrl: Enabling Camera Control for Text-to-Video Generation](https://mlsanity.com/paper/2404.02101/) (2024)
- [I2VGen-XL: High-Quality Image-to-Video Synthesis via Cascaded Diffusion Models](https://mlsanity.com/paper/2311.04145/) (2023)
- [Text-To-4D Dynamic Scene Generation](https://mlsanity.com/paper/2301.11280/) (2023)
- [DepthCrafter: Generating Consistent Long Depth Sequences for Open-world Videos](https://mlsanity.com/paper/2409.02095/) (2024)
- [Align Your Gaussians: Text-to-4D with Dynamic 3D Gaussians and Composed Diffusion Models](https://mlsanity.com/paper/2312.13763/) (2023)
- [Animate124: Animating One Image to 4D Dynamic Scene](https://mlsanity.com/paper/2311.14603/) (2023)
- [4Diffusion: Multi-view Video Diffusion Model for 4D Generation](https://mlsanity.com/paper/2405.20674/) (2024)
- [A Unified Approach for Text- and Image-guided 4D Scene Generation](https://mlsanity.com/paper/2311.16854/) (2023)
- [Geo4D: Leveraging Video Generators for Geometric 4D Scene Reconstruction](https://mlsanity.com/paper/2504.07961/) (2025)
- [DreamScene4D: Dynamic Multi-Object Scene Generation from Monocular Videos](https://mlsanity.com/paper/2405.02280/) (2024)
- [Efficiently Reconstructing Dynamic Scenes One D4RT at a Time](https://mlsanity.com/paper/2512.08924/) (2025)
- [Comp4D: LLM-Guided Compositional 4D Scene Generation](https://mlsanity.com/paper/2403.16993/) (2024)
- [Wan-Move: Motion-controllable Video Generation via Latent Trajectory Guidance](https://mlsanity.com/paper/2512.08765/) (2025)
- [Dynamic Point Maps: A Versatile Representation for Dynamic 3D Reconstruction](https://mlsanity.com/paper/2503.16318/) (2025)
- [GeometryCrafter: Consistent Geometry Estimation for Open-world Videos with Diffusion Priors](https://mlsanity.com/paper/2504.01016/) (2025)
- [Any4D: Unified Feed-Forward Metric 4D Reconstruction](https://mlsanity.com/paper/2512.10935/) (2025)
- [4DNeX: Feed-Forward 4D Generative Modeling Made Easy](https://mlsanity.com/paper/2508.13154/) (2025)
- [V-DPM: 4D Video Reconstruction with Dynamic Point Maps](https://mlsanity.com/paper/2601.09499/) (2026)
- [Trace Anything: Representing Any Video in 4D via Trajectory Fields](https://mlsanity.com/paper/2510.13802/) (2025)
- [4RC: 4D Reconstruction via Conditional Querying Anytime and Anywhere](https://mlsanity.com/paper/2602.10094/) (2026)
- [Taming Video Models for 3D and 4D Generation via Zero-Shot Camera Control](https://mlsanity.com/paper/2509.15130/) (2025)
- [Video Generation Models are General-Purpose Vision Learners](https://mlsanity.com/paper/2607.09024/) (2026)
- [Text-to-3D by Stitching a Multi-view Reconstruction Network to a Video Generator](https://mlsanity.com/paper/2510.13454/) (2025)
- [WorldReel: 4D Video Generation with Consistent Geometry and Motion Modeling](https://mlsanity.com/paper/2512.07821/) (2025)

## Abstract (arXiv)

4D generation synthesizes dynamic 3D scenes from conditions such as text or images. Existing methods either reconstruct generated RGB videos with a separate 4D model or adapt a particular video generator to predict geometry directly. The former suffers from distribution mismatch and error propagation, whereas the latter ties 4D prediction to a specific generator and may require retraining when the generator or conditioning regime changes. We ask whether the final denoised latents of video models that share a variational autoencoder (VAE) can instead provide a reusable interface to explicit 4D prediction. Building on this insight, we introduce direct latent-to-4D generation and instantiate it as Latent-to-4D, which bypasses RGB by aligning a video latent with the token grid of a pretrained 4D decoder and refining it through frame-wise and global spatiotemporal attention. Trained on roughly 1K existing reconstruction clips, a single checkpoint transfers unchanged across multiple video diffusion transformers within the same VAE family. On Text4D-200 and I4D-200, Latent-to-4D surpasses matched same-latent Wan+4RC cascades in projection-based DINO-F1 by 2.88--3.45 and 5.81 points, respectively, while also being preferred by human raters for geometry, temporal stability, and overall quality.