# What Is The Best 3D Scene Representation for Robotics? From Geometric to Foundation Models

arXiv: https://arxiv.org/abs/2512.03422 · HTML: https://mlsanity.com/paper/2512.03422/

## Key facts

- Cited by 29 papers in the map (data through 14 Oct 2026).
- Builds on 254 of its 311 references that are in the map.
- First posted on arXiv 3 Dec 2025.

## Builds on (most cited first, 30 of 254)

- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [3D Gaussian Splatting for Real-Time Radiance Field Rendering](https://mlsanity.com/paper/2308.04079/) (2023)
- [PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation](https://mlsanity.com/paper/1612.00593/) (2016)
- [nuScenes: A multimodal dataset for autonomous driving](https://mlsanity.com/paper/1903.11027/) (2019)
- [PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space](https://mlsanity.com/paper/1706.02413/) (2017)
- [On the Opportunities and Risks of Foundation Models](https://mlsanity.com/paper/2108.07258/) (2021)
- [Instant Neural Graphics Primitives with a Multiresolution Hash Encoding](https://mlsanity.com/paper/2201.05989/) (2022)
- [Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection](https://mlsanity.com/paper/2303.05499/) (2023)
- [Taming Transformers for High-Resolution Image Synthesis](https://mlsanity.com/paper/2012.09841/) (2020)
- [ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenes](https://mlsanity.com/paper/1702.04405/) (2017)
- [ShapeNet: An Information-Rich 3D Model Repository](https://mlsanity.com/paper/1512.03012/) (2015)
- [DeepSDF: Learning Continuous Signed Distance Functions for Shape Representation](https://mlsanity.com/paper/1901.05103/) (2019)
- [Scalability in Perception for Autonomous Driving: Waymo Open Dataset](https://mlsanity.com/paper/1912.04838/) (2019)
- [π0: A Vision-Language-Action Flow Model for General Robot Control](https://mlsanity.com/paper/2410.24164/) (2024)
- [Do As I Can, Not As I Say: Grounding Language in Robotic Affordances](https://mlsanity.com/paper/2204.01691/) (2022)
- [RT-1: Robotics Transformer for Real-World Control at Scale](https://mlsanity.com/paper/2212.06817/) (2022)
- [Mip-NeRF 360: Unbounded Anti-Aliased Neural Radiance Fields](https://mlsanity.com/paper/2111.12077/) (2021)
- [Mip-NeRF: A Multiscale Representation for Anti-Aliasing Neural Radiance Fields](https://mlsanity.com/paper/2103.13415/) (2021)
- [PointPillars: Fast Encoders for Object Detection from Point Clouds](https://mlsanity.com/paper/1812.05784/) (2018)
- [NeuS: Learning Neural Implicit Surfaces by Volume Rendering for Multi-view Reconstruction](https://mlsanity.com/paper/2106.10689/) (2021)
- [Big Bird: Transformers for Longer Sequences](https://mlsanity.com/paper/2007.14062/) (2020)
- [Plenoxels: Radiance Fields without Neural Networks](https://mlsanity.com/paper/2112.05131/) (2021)
- [DUSt3R: Geometric 3D Vision Made Easy](https://mlsanity.com/paper/2312.14132/) (2023)
- [Objaverse: A Universe of Annotated 3D Objects](https://mlsanity.com/paper/2212.08051/) (2022)
- [VGGT: Visual Geometry Grounded Transformer](https://mlsanity.com/paper/2503.11651/) (2025)
- [Matterport3D: Learning from RGB-D Data in Indoor Environments](https://mlsanity.com/paper/1709.06158/) (2017)
- [VoxelNet: End-to-End Learning for Point Cloud Based 3D Object Detection](https://mlsanity.com/paper/1711.06396/) (2017)

## Built on by (most cited first, 29 of 29)

- [GaussianDWM: 3D Gaussian Driving World Model for Unified Scene Understanding and Multi-Modal Generation](https://mlsanity.com/paper/2512.23180/) (2025)
- [Task-Specific Efficiency Analysis: When Small Language Models Outperform Large Language Models](https://mlsanity.com/paper/2603.21389/) (2026)
- [Uni-World VLA: Interleaved World Modeling and Planning for Autonomous Driving](https://mlsanity.com/paper/2603.27287/) (2026)
- [OmniDrive-R1: Reinforcement-driven Interleaved Multi-modal Chain-of-Thought for Trustworthy Vision-Language Autonomous Driving](https://mlsanity.com/paper/2512.14044/) (2025)
- [UniPR-3D: Towards Universal Visual Place Recognition with Visual Geometry Grounded Transformer](https://mlsanity.com/paper/2512.21078/) (2025)
- [Generation Models Know Space: Unleashing Implicit 3D Priors for Scene Understanding](https://mlsanity.com/paper/2603.19235/) (2026)
- [When Models Learn to Ask Why: Adaptive Causal Reasoning for Trustworthy Medical Vision-Language Models](https://mlsanity.com/paper/2603.23085/) (2026)
- [Reloc-VGGT: Visual Re-localization with Geometry Grounded Transformer](https://mlsanity.com/paper/2512.21883/) (2025)
- [CARE What Fails: Contrastive Anchored-REflection for Verifiable Multimodal Reasoning](https://mlsanity.com/paper/2512.19554/) (2025)
- [RAYNOVA: Scale-Temporal Autoregressive World Modeling in Ray Space](https://mlsanity.com/paper/2602.20685/) (2026)
- [Dream-SLAM: Dreaming the Unseen for Active SLAM in Dynamic Environments](https://mlsanity.com/paper/2602.21967/) (2026)
- [Mamba-VGGT: Persistent Long-Sequence Video Geometry Grounded Transformer via External Sliding Window Mamba Memory](https://mlsanity.com/paper/2605.17478/) (2026)
- [ObjSplat: Geometry-Aware Gaussian Surfels for Active Object Reconstruction](https://mlsanity.com/paper/2601.06997/) (2026)
- [WeatherCity: Urban Scene Reconstruction with Controllable Multi-Weather Transformation](https://mlsanity.com/paper/2602.22096/) (2026)
- [SpatialFusion: Endowing Unified Image Generation with Intrinsic 3D Geometric Awareness](https://mlsanity.com/paper/2604.26341/) (2026)
- [VGGT-Occ: Geometry-Grounded and Density-Aware Gated Fusion for 3D Occupancy Prediction](https://mlsanity.com/paper/2605.16911/) (2026)
- [VEOcc: Voxel-Centric Online Semantic Occupancy Prediction For Embodied Scene Understanding](https://mlsanity.com/paper/2605.25059/) (2026)
- [Visual Relocalization from Sparse Views in Aliased and Low-Texture Environments via Novel View Synthesis](https://mlsanity.com/paper/2607.22147/) (2026)
- [DINO-VO: Learning Where to Focus for Enhanced State Estimation](https://mlsanity.com/paper/2604.04055/) (2026)
- [Appearance Decomposition Gaussian Splatting for Multi-Traversal Reconstruction](https://mlsanity.com/paper/2604.05908/) (2026)
- [Geometry-Consistent Endoscopic Representations for Image-Guided Navigation via Structured Foundation Model Adaptation](https://mlsanity.com/paper/2606.17340/) (2026)
- [RoGS: Adaptive Meshgrid Gaussian for Large-Scale Road Surface Mapping](https://mlsanity.com/paper/2607.15048/) (2026)
- [MAGiSt3R: Multi-Agent Feed-forward 3D Reconstruction from Monocular RGB Videos](https://mlsanity.com/paper/2607.15211/) (2026)
- [SLAM in Low-Light Environments: Project Report](https://mlsanity.com/paper/2607.17699/) (2026)
- [MOJITO: Modal Joint Learning for Unified End-to-End Autonomous Driving](https://mlsanity.com/paper/2607.23511/) (2026)
- [Multi-Submap Implicit Neural SLAM with Local-to-Global Loop Closure for Large-Scale Scene Reconstruction](https://mlsanity.com/paper/2608.09146/) (2026)
- [GaussianDWM++: Language-Grounded 3D Gaussian Driving World Model for Unified Scene Understanding, Editing, and Multi-Modal Generation](https://mlsanity.com/paper/2608.16234/) (2026)
- [Toward Unified Robot Learning: Bridging Representation, Vision-Language-Action, and World Models](https://mlsanity.com/paper/2609.03927/) (2026)
- [Adaptive World Memory 3D Foundation Model for Scalable 3D Mapping, Localization, and Rendering](https://mlsanity.com/paper/2609.21502/) (2026)

## Abstract (arXiv)

In this paper, we provide a comprehensive overview of existing scene representation methods for robotics, covering traditional representations such as point clouds, voxels, signed distance functions (SDF), and scene graphs, as well as more recent neural representations like Neural Radiance Fields (NeRF), 3D Gaussian Splatting (3DGS), and the emerging Foundation Models. While current SLAM and localization systems predominantly rely on sparse representations like point clouds and voxels, dense scene representations are expected to play a critical role in downstream tasks such as navigation and obstacle avoidance. Moreover, neural representations such as NeRF, 3DGS, and foundation models are well-suited for integrating high-level semantic features and language-based priors, enabling more comprehensive 3D scene understanding and embodied intelligence. In this paper, we categorized the core modules of robotics into five parts (Perception, Mapping, Localization, Navigation, Manipulation). We start by presenting the standard formulation of different scene representation methods and comparing the advantages and disadvantages of scene representation across different modules. This survey is centered around the question: What is the best 3D scene representation for robotics? We then discuss the future development trends of 3D scene representations, with a particular focus on how the 3D Foundation Model could replace current methods as the unified solution for future robotic applications. The remaining challenges in fully realizing this model are also explored. We aim to offer a valuable resource for both newcomers and experienced researchers to explore the future of 3D scene representations and their application in robotics. We have published an open-source project on GitHub and will continue to add new works and technologies to this project.