# POMA-3D: The Point Map Way to 3D Scene Understanding

arXiv: https://arxiv.org/abs/2511.16567 · HTML: https://mlsanity.com/paper/2511.16567/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 51 of its 60 references that are in the map.
- First posted on arXiv 20 Nov 2025.

## Builds on (most cited first, 30 of 51)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Qwen2.5-VL Technical Report](https://mlsanity.com/paper/2502.13923/) (2025)
- [PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation](https://mlsanity.com/paper/1612.00593/) (2016)
- [ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenes](https://mlsanity.com/paper/1702.04405/) (2017)
- [LLaVA-OneVision: Easy Visual Task Transfer](https://mlsanity.com/paper/2408.03326/) (2024)
- [InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Models](https://mlsanity.com/paper/2504.10479/) (2025)
- [DUSt3R: Geometric 3D Vision Made Easy](https://mlsanity.com/paper/2312.14132/) (2023)
- [VGGT: Visual Geometry Grounded Transformer](https://mlsanity.com/paper/2503.11651/) (2025)
- [Point Transformer](https://mlsanity.com/paper/2011.00931/) (2020)
- [Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecture](https://mlsanity.com/paper/2301.08243/) (2023)
- [Grounding Image Matching in 3D with MASt3R](https://mlsanity.com/paper/2406.09756/) (2024)
- [A Point Set Generation Network for 3D Object Reconstruction from a Single Image](https://mlsanity.com/paper/1612.00603/) (2016)
- [V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planning](https://mlsanity.com/paper/2506.09985/) (2025)
- [Habitat-Matterport 3D Dataset (HM3D): 1000 Large-scale 3D Environments for Embodied AI](https://mlsanity.com/paper/2109.08238/) (2021)
- [ARKitScenes: A Diverse Real-World Dataset For 3D Indoor Scene Understanding Using Mobile RGB-D Data](https://mlsanity.com/paper/2111.08897/) (2021)
- [ScanRefer: 3D Object Localization in RGB-D Scans using Natural Language](https://mlsanity.com/paper/1912.08830/) (2019)
- [ProcTHOR: Large-Scale Embodied AI Using Procedural Generation](https://mlsanity.com/paper/2206.06994/) (2022)
- [An Embodied Generalist Agent in 3D World](https://mlsanity.com/paper/2311.12871/) (2023)
- [ScanQA: 3D Question Answering for Spatial Scene Understanding](https://mlsanity.com/paper/2112.10482/) (2021)
- [PointGroup: Dual-Set Point Grouping for 3D Instance Segmentation](https://mlsanity.com/paper/2004.01658/) (2020)
- [SQA3D: Situated Question Answering in 3D Scenes](https://mlsanity.com/paper/2210.07474/) (2022)
- [Mask3D: Mask Transformer for 3D Semantic Instance Segmentation](https://mlsanity.com/paper/2210.03105/) (2022)
- [Structured3D: A Large Photo-realistic Dataset for Structured 3D Modeling](https://mlsanity.com/paper/1908.00222/) (2019)
- [ULIP: Learning a Unified Representation of Language, Images, and Point Clouds for 3D Understanding](https://mlsanity.com/paper/2212.05171/) (2022)
- [A Survey of Embodied AI: From Simulators to Research Tasks](https://mlsanity.com/paper/2103.04918/) (2021)
- [Uni3D: Exploring Unified 3D Representation at Scale](https://mlsanity.com/paper/2310.06773/) (2023)
- [3D-SIS: 3D Semantic Instance Segmentation of RGB-D Scans](https://mlsanity.com/paper/1812.07003/) (2018)
- [LL3DA: Visual Interactive Instruction Tuning for Omni-3D Understanding, Reasoning, and Planning](https://mlsanity.com/paper/2311.18651/) (2023)
- [3D-VisTA: Pre-trained Transformer for 3D Vision and Text Alignment](https://mlsanity.com/paper/2308.04352/) (2023)
- [ULIP-2: Towards Scalable Multimodal Pre-training for 3D Understanding](https://mlsanity.com/paper/2305.08275/) (2023)

## Built on by (most cited first, 4 of 4)

- [From None to All: Self-Supervised 3D Reconstruction via Novel View Synthesis](https://mlsanity.com/paper/2603.27455/) (2026)
- [RGB-Pointmap Pretraining for Unified 3D Scene Understanding](https://mlsanity.com/paper/2604.02546/) (2026)
- [SR-JEPA: Learning Predictive Latent State in 3D Scenes](https://mlsanity.com/paper/2608.05774/) (2026)
- [RDGSplat: Render-Dedicated Geometry for Novel View Synthesis](https://mlsanity.com/paper/2610.09173/) (2026)

## Abstract (arXiv)

In this paper, we introduce POMA-3D, the first self-supervised 3D representation model learned from point maps. Point maps encode explicit 3D coordinates on a structured 2D grid, preserving global 3D geometry while remaining compatible with the input format of 2D foundation models. To transfer rich 2D priors into POMA-3D, a view-to-scene alignment strategy is designed. Moreover, as point maps are view-dependent with respect to a canonical space, we introduce POMA-JEPA, a joint embedding-predictive architecture that enforces geometrically consistent point map features across multiple views. Additionally, we introduce ScenePoint, a point map dataset constructed from 6.5K room-level RGB-D scenes and 1M 2D image scenes to facilitate large-scale POMA-3D pretraining. Experiments show that POMA-3D serves as a strong backbone for both specialist and generalist 3D understanding. It benefits diverse tasks, including 3D question answering, embodied navigation, scene retrieval, and embodied localization, all achieved using only geometric inputs (i.e., 3D coordinates). Overall, our POMA-3D explores a point map way to 3D scene understanding, addressing the scarcity of pretrained priors and limited data in 3D representation learning. Project Page: https://matchlab-imperial.github.io/poma3d/