# SPA: 3D Spatial-Awareness Enables Effective Embodied Representation

arXiv: https://arxiv.org/abs/2410.08208 · HTML: https://mlsanity.com/paper/2410.08208/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- Builds on 75 of its 85 references that are in the map.
- First posted on arXiv 10 Oct 2024.

## Builds on (most cited first, 30 of 75)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [Gaussian Error Linear Units (GELUs)](https://mlsanity.com/paper/1606.08415/) (2016)
- [Deformable DETR: Deformable Transformers for End-to-End Object Detection](https://mlsanity.com/paper/2010.04159/) (2020)
- [ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenes](https://mlsanity.com/paper/1702.04405/) (2017)
- [Diffusion Policy: Visuomotor Policy Learning via Action Diffusion](https://mlsanity.com/paper/2303.04137/) (2023)
- [InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks](https://mlsanity.com/paper/2312.14238/) (2023)
- [Improved Baselines with Momentum Contrastive Learning](https://mlsanity.com/paper/2003.04297/) (2020)
- [Decoupled Weight Decay Regularization](https://mlsanity.com/paper/1711.05101/) (2017)
- [NeuS: Learning Neural Implicit Surfaces by Volume Rendering for Multi-view Reconstruction](https://mlsanity.com/paper/2106.10689/) (2021)
- [An Empirical Study of Training Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.02057/) (2021)
- [Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardware](https://mlsanity.com/paper/2304.13705/) (2023)
- [Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-shot Cross-dataset Transfer](https://mlsanity.com/paper/1907.01341/) (2019)
- [LIBERO: Benchmarking Knowledge Transfer for Lifelong Robot Learning](https://mlsanity.com/paper/2306.03310/) (2023)
- [VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Training](https://mlsanity.com/paper/2203.12602/) (2022)
- [BEVFormer: Learning Bird's-Eye-View Representation from Multi-Camera Images via Spatiotemporal Transformers](https://mlsanity.com/paper/2203.17270/) (2022)
- [Real-Time Single Image and Video Super-Resolution Using an Efficient Sub-Pixel Convolutional Neural Network](https://mlsanity.com/paper/1609.05158/) (2016)
- [Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learning](https://mlsanity.com/paper/1910.10897/) (2019)
- [PlenOctrees for Real-time Rendering of Neural Radiance Fields](https://mlsanity.com/paper/2103.14024/) (2021)
- [DROID: A Large-Scale In-The-Wild Robot Manipulation Dataset](https://mlsanity.com/paper/2403.12945/) (2024)
- [ScanNet++: A High-Fidelity Dataset of 3D Indoor Scenes](https://mlsanity.com/paper/2308.11417/) (2023)
- [Implicit Geometric Regularization for Learning Shapes](https://mlsanity.com/paper/2002.10099/) (2020)
- [SpatialVLM: Endowing Vision-Language Models with Spatial Reasoning Capabilities](https://mlsanity.com/paper/2401.12168/) (2024)
- [R3M: A Universal Visual Representation for Robot Manipulation](https://mlsanity.com/paper/2203.12601/) (2022)
- [EVA: Exploring the Limits of Masked Visual Representation Learning at Scale](https://mlsanity.com/paper/2211.07636/) (2022)

## Built on by (most cited first, 28 of 28)

- [Aether: Geometric-Aware Unified World Modeling](https://mlsanity.com/paper/2503.18945/) (2025)
- [StreamForest: Efficient Online Video Understanding with Persistent Event Memory](https://mlsanity.com/paper/2509.24871/) (2025)
- [VQ-VLA: Improving Vision-Language-Action Models via Scaling Vector-Quantized Action Tokenizers](https://mlsanity.com/paper/2507.01016/) (2025)
- [MindJourney: Test-Time Scaling with World Models for Spatial Reasoning](https://mlsanity.com/paper/2507.12508/) (2025)
- [Lift3D Foundation Policy: Lifting 2D Large-Scale Pretrained Models for Robust 3D Robotic Manipulation](https://mlsanity.com/paper/2411.18623/) (2024)
- [CoMo: Learning Continuous Latent Motion from Internet Videos for Scalable Robot Learning](https://mlsanity.com/paper/2505.17006/) (2025)
- [DeepVerse: 4D Autoregressive Video Generation as a World Model](https://mlsanity.com/paper/2506.01103/) (2025)
- [FantasyWorld: Geometry-Consistent World Modeling via Unified Video and 3D Prediction](https://mlsanity.com/paper/2509.21657/) (2025)
- [Feat2GS: Probing Visual Foundation Models with Gaussian Splatting](https://mlsanity.com/paper/2412.09606/) (2024)
- [Seed3D 1.0: From Images to High-Fidelity Simulation-Ready 3D Assets](https://mlsanity.com/paper/2510.19944/) (2025)
- [GP3: A 3D Geometry-Aware Policy with Multi-View Images for Robotic Manipulation](https://mlsanity.com/paper/2509.15733/) (2025)
- [ManiGaussian++: General Robotic Bimanual Manipulation with Hierarchical Gaussian World Model](https://mlsanity.com/paper/2506.19842/) (2025)
- [Rethinking Latent Redundancy in Behavior Cloning: An Information Bottleneck Approach for Robot Manipulation](https://mlsanity.com/paper/2502.02853/) (2025)
- [Neural Brain: A Neuroscience-inspired Framework for Embodied Agents](https://mlsanity.com/paper/2505.07634/) (2025)
- [EmbodiedMAE: A Unified 3D Multi-Modal Representation for Robot Manipulation](https://mlsanity.com/paper/2505.10105/) (2025)
- [Action-Geometry Prediction with 3D Geometric Prior for Bimanual Manipulation](https://mlsanity.com/paper/2602.23814/) (2026)
- [Perspective from a Higher Dimension: Can 3D Geometric Priors Help Visual Floorplan Localization?](https://mlsanity.com/paper/2507.18881/) (2025)
- [Perspective from a Broader Context: Can Room Style Knowledge Help Visual Floorplan Localization?](https://mlsanity.com/paper/2508.01216/) (2025)
- [WAT: Online Video Understanding Needs Watching Before Thinking](https://mlsanity.com/paper/2603.13412/) (2026)
- [Lift3D-VLA: Lifting VLA Models to 3D Geometry and Dynamics-Aware Manipulation](https://mlsanity.com/paper/2607.06564/) (2026)
- [Boosting Robotic Manipulation Generalization with Minimal Costly Data](https://mlsanity.com/paper/2503.19516/) (2025)
- [CLAR: Learning 3D Representations for Robotic Manipulation by Fusing Masked Reconstruction with Multi-Level Contrastive Alignment](https://mlsanity.com/paper/2507.08262/) (2025)
- [SpaRRTa: A Synthetic Benchmark for Evaluating Spatial Intelligence in Visual Foundation Models](https://mlsanity.com/paper/2601.11729/) (2026)
- [Learning 3D Representations for Spatial Intelligence from Unposed Multi-View Images](https://mlsanity.com/paper/2604.10573/) (2026)
- [Learning Structural Latent Points for Efficient Visual Representations in Robotic Manipulation](https://mlsanity.com/paper/2605.21258/) (2026)
- [Which Pretraining Paradigm Better Serves Spatial Intelligence? An Empirical Comparison of Vision-Language and Video Generation Models](https://mlsanity.com/paper/2605.28132/) (2026)
- [The Moving Eye: Enhancing VLA Spatial Generalization via Hybrid Dynamic Data Collection](https://mlsanity.com/paper/2607.02322/) (2026)
- [Space as an Interventional Invariant: Cross-Modal Predictive Geometry for Stratified Cities and Em-Spaced Intelligence](https://mlsanity.com/paper/2609.11959/) (2026)

## Abstract (arXiv)

In this paper, we introduce SPA, a novel representation learning framework that emphasizes the importance of 3D spatial awareness in embodied AI. Our approach leverages differentiable neural rendering on multi-view images to endow a vanilla Vision Transformer (ViT) with intrinsic spatial understanding. We present the most comprehensive evaluation of embodied representation learning to date, covering 268 tasks across 8 simulators with diverse policies in both single-task and language-conditioned multi-task scenarios. The results are compelling: SPA consistently outperforms more than 10 state-of-the-art representation methods, including those specifically designed for embodied AI, vision-centric tasks, and multi-modal applications, while using less training data. Furthermore, we conduct a series of real-world experiments to confirm its effectiveness in practical scenarios. These results highlight the critical role of 3D spatial awareness for embodied representation learning. Our strongest model takes more than 6000 GPU hours to train and we are committed to open-sourcing all code and model weights to foster future research in embodied representation learning. Project Page: https://haoyizhu.github.io/spa/.