# SceneAE: Scene Generation and Reconstruction in Queryable 3D-Anchored Latent Space

arXiv: https://arxiv.org/abs/2610.19708 · HTML: https://mlsanity.com/paper/2610.19708/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Video-generation world models and simulators, one of 11 directions in World models.
- Builds on 28 of its 33 references that are in the map.
- First posted on arXiv 14 Oct 2026.

## Placed in

- World models / [Video-generation world models and simulators](https://mlsanity.com/wm/video-generation-world-models-and-simulators/)

## Builds on (most cited first, 28 of 28)

- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [3D Gaussian Splatting for Real-Time Radiance Field Rendering](https://mlsanity.com/paper/2308.04079/) (2023)
- [ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenes](https://mlsanity.com/paper/1702.04405/) (2017)
- [Wan: Open and Advanced Large-Scale Video Generative Models](https://mlsanity.com/paper/2503.20314/) (2025)
- [CogVideoX: Text-to-Video Diffusion Models with An Expert Transformer](https://mlsanity.com/paper/2408.06072/) (2024)
- [VBench: Comprehensive Benchmark Suite for Video Generative Models](https://mlsanity.com/paper/2311.17982/) (2023)
- [VGGT: Visual Geometry Grounded Transformer](https://mlsanity.com/paper/2503.11651/) (2025)
- [TensoRF: Tensorial Radiance Fields](https://mlsanity.com/paper/2203.09517/) (2022)
- [CogVideo: Large-scale Pretraining for Text-to-Video Generation via Transformers](https://mlsanity.com/paper/2205.15868/) (2022)
- [Depth Anything 3: Recovering the Visual Space from Any Views](https://mlsanity.com/paper/2511.10647/) (2025)
- [Open-Sora: Democratizing Efficient Video Production for All](https://mlsanity.com/paper/2412.20404/) (2024)
- [DL3DV-10K: A Large-Scale Scene Dataset for Deep Learning-based 3D Vision](https://mlsanity.com/paper/2312.16256/) (2023)
- [GS-LRM: Large Reconstruction Model for 3D Gaussian Splatting](https://mlsanity.com/paper/2404.19702/) (2024)
- [GEN3C: 3D-Informed World-Consistent Video Generation with Precise Camera Control](https://mlsanity.com/paper/2503.03751/) (2025)
- [GeoNeRF: Generalizing NeRF with Geometry Priors](https://mlsanity.com/paper/2111.13539/) (2021)
- [LVSM: A Large View Synthesis Model with Minimal 3D Inductive Bias](https://mlsanity.com/paper/2410.17242/) (2024)
- [MVSplat360: Feed-Forward 360 Scene Synthesis from Sparse Views](https://mlsanity.com/paper/2411.04924/) (2024)
- [Cameras as Relative Positional Encoding](https://mlsanity.com/paper/2507.10496/) (2025)
- [Voyager: Long-Range and World-Consistent Video Diffusion for Explorable 3D Scene Generation](https://mlsanity.com/paper/2506.04225/) (2025)
- [Aether: Geometric-Aware Unified World Modeling](https://mlsanity.com/paper/2503.18945/) (2025)
- [Geometry Forcing: Marrying Video Diffusion and 3D Representation for Consistent World Modeling](https://mlsanity.com/paper/2507.07982/) (2025)
- [RayZer: A Self-supervised Large View Synthesis Model](https://mlsanity.com/paper/2505.00702/) (2025)
- [WorldMirror: Universal 3D World Reconstruction with Any-Prior Prompting](https://mlsanity.com/paper/2510.10726/) (2025)
- [World-consistent Video Diffusion with Explicit 3D Modeling](https://mlsanity.com/paper/2412.01821/) (2024)
- [Gen3R: 3D Scene Generation Meets Feed-Forward Reconstruction](https://mlsanity.com/paper/2601.04090/) (2026)
- [Latent Spatial Memory for Video World Models](https://mlsanity.com/paper/2606.09828/) (2026)
- [Repurposing Geometric Foundation Models for Multi-view Diffusion](https://mlsanity.com/paper/2603.22275/) (2026)
- [SceneTok: A Compressed, Diffusable Token Space for 3D Scenes](https://mlsanity.com/paper/2602.18882/) (2026)

## Abstract (arXiv)

Recent video world models have increasingly adopted 3D memory mechanisms to integrate historical observations into a lightweight and persistent scene representation. This naturally raises a complementary question: if 3D representations can serve as an effective mechanism for memorizing observed scenes, can they also be directly used as the latent space for reconstructing and generating scenes? Motivated by this idea, we represent a scene as a compact and unstructured 3D-anchored latent space (SceneAE). Each latent combines an explicit 3D position with an implicit content feature and is spatially grounded to describe its local scene region, forming a sparse latent point cloud. Unlike conventional video-VAE latents that describe how a scene appears from particular viewpoints, our representation directly describes the scene itself, making it naturally applicable to both reconstruction and generation. For reconstruction, once the scene has been generated, our persistent 3D representation can be projected and decoded into RGB-D observations by different target cameras, without the need to regenerate a video for each new camera trajectory. For generation, we leverage the geometry-content separation to first generate camera-controlled 3D geometry and then synthesize content conditioned on it, reducing camera-appearance entanglement and improving camera controllability. Experiments show that this 3D latent achieves superior generation quality, computational cost, and camera controllability.