mlsanityData to 7 Oct 2026

BIP3D: Bridging 2D Images and 3D Perception for Embodied Intelligence

arXiv 2411.14869 ↗First posted 22 Nov 2024cs.CV cs.AI cs.LGcited by 10

Not on the map. Why?

Abstract (arXiv)

In embodied intelligence systems, a key component is 3D perception algorithm, which enables agents to understand their surrounding environments. Previous algorithms primarily rely on point cloud, which, despite offering precise geometric information, still constrain perception performance due to inherent sparsity, noise, and data scarcity. In this work, we introduce a novel image-centric 3D perception model, BIP3D, which leverages expressive image features with explicit 3D position encoding to overcome the limitations of point-centric methods. Specifically, we leverage pre-trained 2D vision foundation models to enhance semantic understanding, and introduce a spatial enhancer module to improve spatial understanding. Together, these modules enable BIP3D to achieve multi-view, multi-modal feature fusion and end-to-end 3D perception. In our experiments, BIP3D outperforms current state-of-the-art results on the EmbodiedScan benchmark, achieving improvements of 5.69% in the 3D detection task and 15.25% in the 3D visual grounding task.

Key facts. Cited by 10 papers in the map (data through 7 Oct 2026). Builds on 41 of its 49 references that are in the map. First posted on arXiv 22 Nov 2024.

Builds on

41 of 49 references.

References · 41

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,764
  3. 2020End-to-End Object Detection with Transformerscited by 7,384
  4. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  5. 2017PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Spacecited by 4,384
  6. 2020Deformable DETR: Deformable Transformers for End-to-End Object Detectioncited by 3,410
  7. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,274
  8. 2017ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenescited by 3,088
  9. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,360
  10. 2021An Empirical Study of Training Self-Supervised Vision Transformerscited by 1,610
  11. 2017Matterport3D: Learning from RGB-D Data in Indoor Environmentscited by 1,354
  12. 2022DINO: DETR with Improved DeNoising Anchor Boxes for End-to-End Object Detectioncited by 1,306
29 more
  1. 20194D Spatio-Temporal ConvNets: Minkowski Convolutional Neural Networkscited by 1,201
  2. 2021Grounded Language-Image Pre-trainingcited by 1,147
  3. 2021MDETR -- Modulated Detection for End-to-End Multi-Modal Understandingcited by 811
  4. 2023EVA-CLIP: Improved Training Techniques for CLIP at Scalecited by 786
  5. 2023Scaling Vision Transformers to 22 Billion Parameterscited by 690
  6. 2019Deep Hough Voting for 3D Object Detection in Point Cloudscited by 648
  7. 2021RegionCLIP: Region-based Language-Image Pretrainingcited by 612
  8. 2022PETR: Position Embedding Transformation for Multi-View 3D Object Detectioncited by 550
  9. 2021PointCLIP: Point Cloud Understanding by CLIPcited by 461
  10. 2019ScanRefer: 3D Object Localization in RGB-D Scans using Natural Languagecited by 446
  11. 2023EVA-02: A Visual Representation for Neon Genesiscited by 377
  12. 2023ConceptFusion: Open-set Multimodal 3D Mappingcited by 302
  13. 2023Uni3D: Exploring Unified 3D Representation at Scalecited by 231
  14. 2021Group-Free 3D Object Detection via Transformerscited by 215
  15. 2019RIO: 3D Object Instance Re-Localization in Changing Indoor Environmentscited by 187
  16. 2022Sparse4D: Multi-view 3D Object Detection with Sparse Spatial-Temporal Fusioncited by 161
  17. 2022CLIP2Point: Transfer CLIP to Point Cloud Classification with Image-Depth Pre-trainingcited by 159
  18. 2021ImVoxelNet: Image to Voxels Projection for Monocular and Multi-View General-Purpose 3D Object Detectioncited by 157
  19. 2023EmbodiedScan: A Holistic Multi-Modal 3D Perception Suite Towards Embodied AIcited by 146
  20. 2021Bottom Up Top Down Detection Transformers for Language Grounding in Images and Point Cloudscited by 128
  21. 2021SAT: 2D Semantics Assisted Training for 3D Visual Groundingcited by 124
  22. 2020ImVoteNet: Boosting 3D Object Detection in Point Clouds with Image Votescited by 119
  23. 20223D-SPS: Single-Stage 3D Visual Grounding via Referred Point Progressive Selectioncited by 109
  24. 2021FCAF3D: Fully Convolutional Anchor-Free 3D Object Detectioncited by 97
  25. 2021LanguageRefer: Spatial-Language Model for 3D Visual Groundingcited by 88
  26. 2023NeRF-Det: Learning Geometry-Aware Volumetric Representation for Multi-View 3D Object Detectioncited by 51
  27. 2024UniDet3D: Multi-dataset Indoor 3D Object Detectioncited by 20
  28. 2023Dense Object Grounding in 3D Scenescited by 15
  29. 2023Cross3DVG: Cross-Dataset 3D Visual Grounding on Different RGB-D Scanscited by 10

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 3; 2026: 7.

10 papers.

  1. 2026HoloBrain-0 Technical Reportcited by 20
  2. 2025SEM: Enhancing Spatial Understanding for Robust Robot Manipulationcited by 8
  3. 2025DEGround: An Effective Baseline for Ego-centric 3D Visual Grounding with a Homogeneous Frameworkcited by 4
  4. 2026OpenSpatial: A Principled Data Engine for Empowering Spatial Intelligencecited by 4
  5. 2025P3: Toward Versatile Embodied Agentscited by 2
  6. 2026HoloAgent-0: A Unified Embodied Agent Framework with 3D Spatial Memorycited by 1
  7. 2026G-ray: Ray-Level Relative Geometric Position Encoding in Multi-View Vision Transformers under Camera Heterogeneitycited by 1
  8. 2026Which Pretraining Paradigm Better Serves Spatial Intelligence? An Empirical Comparison of Vision-Language and Video Generation Modelscited by 0
  9. 2026SeqAlign3DVG: A Sequence-Aligned Benchmark and Voxel Reasoning Framework for 3D Visual Groundingcited by 0
  10. 2026Learning Semantic Inpainting for Animatable Gaussian Head Avatarscited by 0