mlsanityData to 7 Oct 2026

Locate 3D: Real-World Object Localization via Self-Supervised Learning in 3D

arXiv 2504.14151 ↗First posted 19 Apr 2025cs.CV cs.AI cs.ROcited by 23

Not on the map. Why?

Abstract (arXiv)

We present LOCATE 3D, a model for localizing objects in 3D scenes from referring expressions like "the small coffee table between the sofa and the lamp." LOCATE 3D sets a new state-of-the-art on standard referential grounding benchmarks and showcases robust generalization capabilities. Notably, LOCATE 3D operates directly on sensor observation streams (posed RGB-D frames), enabling real-world deployment on robots and AR devices. Key to our approach is 3D-JEPA, a novel self-supervised learning (SSL) algorithm applicable to sensor point clouds. It takes as input a 3D pointcloud featurized using 2D foundation models (CLIP, DINO). Subsequently, masked prediction in latent space is employed as a pretext task to aid the self-supervised learning of contextualized pointcloud features. Once trained, the 3D-JEPA encoder is finetuned alongside a language-conditioned decoder to jointly predict 3D masks and bounding boxes. Additionally, we introduce LOCATE 3D DATASET, a new dataset for 3D referential grounding, spanning multiple capture setups with over 130K annotations. This enables a systematic study of generalization capabilities as well as a stronger model.

Key facts. Cited by 23 papers in the map (data through 7 Oct 2026). Builds on 60 of its 67 references that are in the map. First posted on arXiv 19 Apr 2025.

Builds on

60 of 67 references.

References · 60

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2024The Llama 3 Herd of Modelscited by 14,192
  4. 2023Segment Anythingcited by 8,158
  5. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,764
  6. 2020End-to-End Object Detection with Transformerscited by 7,384
  7. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  8. 2016PointNet: Deep Learning on Point Sets for 3D Classification and Segmentationcited by 5,223
  9. 2024GPT-4o System Cardcited by 4,467
  10. 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,346
  11. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,274
  12. 2017ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenescited by 3,088
38 more
  1. 2024SAM 2: Segment Anything in Images and Videoscited by 2,784
  2. 2015VQA: Visual Question Answeringcited by 2,489
  3. 2018Deep contextualized word representationscited by 2,285
  4. 20163D U-Net: Learning Dense Volumetric Segmentation from Sparse Annotationcited by 1,471
  5. 2020Point Transformercited by 1,462
  6. 2017Deep Setscited by 1,394
  7. 2018Averaging Weights Leads to Wider Optima and Better Generalizationcited by 1,194
  8. 2021Per-Pixel Classification is Not All You Need for Semantic Segmentationcited by 1,148
  9. 2023Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecturecited by 982
  10. 2021MDETR -- Modulated Detection for End-to-End Multi-Modal Understandingcited by 811
  11. 2023ScanNet++: A High-Fidelity Dataset of 3D Indoor Scenescited by 747
  12. 2022Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distributioncited by 722
  13. 2021Point-BERT: Pre-training 3D Point Cloud Transformers with Masked Point Modelingcited by 647
  14. 2023Point Transformer V3: Simpler, Faster, Strongercited by 579
  15. 2022OpenScene: 3D Scene Understanding with Open Vocabulariescited by 556
  16. 2020PointContrast: Unsupervised Pre-training for 3D Point Cloud Understandingcited by 502
  17. 2024Revisiting Feature Prediction for Learning Visual Representations from Videocited by 490
  18. 2021PointCLIP: Point Cloud Understanding by CLIPcited by 461
  19. 2022Masked Autoencoders for Point Cloud Self-supervised Learningcited by 456
  20. 2019ScanRefer: 3D Object Localization in RGB-D Scans using Natural Languagecited by 446
  21. 2021ARKitScenes: A Diverse Real-World Dataset For 3D Indoor Scene Understanding Using Mobile RGB-D Datacited by 442
  22. 2022Decomposing NeRF for Editing via Feature Field Distillationcited by 370
  23. 2023An Embodied Generalist Agent in 3D Worldcited by 355
  24. 2023Recognize Anything: A Strong Image Tagging Modelcited by 305
  25. 2022Simple Open-Vocabulary Object Detection with Vision Transformerscited by 303
  26. 2023ConceptFusion: Open-set Multimodal 3D Mappingcited by 302
  27. 2023Set-of-Mark Prompting Unleashes Extraordinary Visual Grounding in GPT-4Vcited by 291
  28. 2022Point-M2AE: Multi-scale Masked Autoencoders for Hierarchical Point Cloud Pre-trainingcited by 263
  29. 2021Self-Supervised Pretraining of 3D Features on any Point-Cloudcited by 217
  30. 20233D-VisTA: Pre-trained Transformer for 3D Vision and Text Alignmentcited by 201
  31. 2021InstanceRefer: Cooperative Holistic Understanding for Visual Grounding on Point Clouds through Instance Multi-level Contextual Referringcited by 140
  32. 2023LLM-Grounder: Open-Vocabulary 3D Visual Grounding with Large Language Model as an Agentcited by 134
  33. 2021Bottom Up Top Down Detection Transformers for Language Grounding in Images and Point Cloudscited by 128
  34. 2021SAT: 2D Semantics Assisted Training for 3D Visual Groundingcited by 124
  35. 2022Language Conditioned Spatial Relation Reasoning for 3D Object Groundingcited by 121
  36. 2023RoboVQA: Multimodal Long-Horizon Reasoning for Roboticscited by 116
  37. 20223D-SPS: Single-Stage 3D Visual Grounding via Referred Point Progressive Selectioncited by 109
  38. 2021LanguageRefer: Spatial-Language Model for 3D Visual Groundingcited by 88

50 most cited of 60 All 60 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 10; 2026: 13.

23 papers.

Sort:

By area:

  1. 2025Concerto: Joint 2D-3D Self-Supervised Learning Emerges Spatial Representationscited by 42
  2. 2025Particulate: Feed-Forward 3D Object Articulationcited by 20
  3. 2025Reasoning in Space via Grounding in the Worldcited by 18
  4. 2025PlaceIt3D: Language-Guided Object Placement in Real 3D Scenescited by 8
  5. 2026REST3D: Reconstructing Physically Stable 3D Scenes from a Single Imagecited by 6
  6. 2025FoundationMotion: Auto-Labeling and Reasoning about Spatial Movement in Videoscited by 5
  7. 2025CrossJEPA: Cross-Modal Joint-Embedding Predictive Architecture for Efficient 3D Representation Learning from 2D Imagescited by 4
  8. 2025SegDINO3D: 3D Instance Segmentation Empowered by Both Image-Level and Object-Level 2D Featurescited by 3
  9. 2026Lifelong Embodied Navigation Learningcited by 3
  10. 2025Unlocking 3D Affordance Segmentation with 2D Semantic Knowledgecited by 2
  11. 2026T-FunS3D: Task-Driven Hierarchical Open-Vocabulary 3D Functionality Segmentationcited by 2
  12. 2026Whareformer: Learning to Track What is Where in Long Egocentric Videoscited by 2
11 more
  1. 2026Qwen-3D: A Generalist 3D Vision-Language Model for Spatial Understandingcited by 2
  2. 2025M3DMap: Object-aware Multimodal 3D Mapping for Dynamic Environmentscited by 1
  3. 2025Large Pre-Trained Models for Bimanual Manipulation in 3Dcited by 1
  4. 2026Memory Over Maps: 3D Object Localization Without Reconstructioncited by 1
  5. 2026RGB-Pointmap Pretraining for Unified 3D Scene Understandingcited by 0
  6. 2026Open-Vocabulary and Referring Segmentation for 3D Gaussians Using 2D Detectorscited by 0
  7. 2026G2TAM: Geometry Grounded Track Anything Modelcited by 0
  8. 2026SR-JEPA: Learning Predictive Latent State in 3D Scenescited by 0
  9. 2026Weave: Learning Whole-Body Dexterous Loco-Manipulation from Human-Object Interactionscited by 0
  10. 2026Toward Comprehensive 3D Grounding: Orientation Grounding through Vision-Language Modelscited by 0
  11. 2026HIGS: Hierarchical Implicit Grids for Joint Geometric and Semantic Scene Understandingcited by 0