# TriCLIP-3D: A Unified Parameter-Efficient Framework for Tri-Modal 3D Visual Grounding based on CLIP

arXiv: https://arxiv.org/abs/2507.14904 · HTML: https://mlsanity.com/paper/2507.14904/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 34 of its 45 references that are in the map.
- First posted on arXiv 20 Jul 2025.

## Builds on (most cited first, 30 of 34)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Squeeze-and-Excitation Networks](https://mlsanity.com/paper/1709.01507/) (2017)
- [PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space](https://mlsanity.com/paper/1706.02413/) (2017)
- [ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenes](https://mlsanity.com/paper/1702.04405/) (2017)
- [Decoupled Weight Decay Regularization](https://mlsanity.com/paper/1711.05101/) (2017)
- [4D Spatio-Temporal ConvNets: Minkowski Convolutional Neural Networks](https://mlsanity.com/paper/1904.08755/) (2019)
- [Deep Hough Voting for 3D Object Detection in Point Clouds](https://mlsanity.com/paper/1904.09664/) (2019)
- [PointCLIP: Point Cloud Understanding by CLIP](https://mlsanity.com/paper/2112.02413/) (2021)
- [ScanRefer: 3D Object Localization in RGB-D Scans using Natural Language](https://mlsanity.com/paper/1912.08830/) (2019)
- [SLIP: Self-supervision meets Language-Image Pre-training](https://mlsanity.com/paper/2112.12750/) (2021)
- [PointGroup: Dual-Set Point Grouping for 3D Instance Segmentation](https://mlsanity.com/paper/2004.01658/) (2020)
- [ULIP: Learning a Unified Representation of Language, Images, and Point Clouds for 3D Understanding](https://mlsanity.com/paper/2212.05171/) (2022)
- [ULIP-2: Towards Scalable Multimodal Pre-training for 3D Understanding](https://mlsanity.com/paper/2305.08275/) (2023)
- [RIO: 3D Object Instance Re-Localization in Changing Indoor Environments](https://mlsanity.com/paper/1908.06109/) (2019)
- [LAMM: Language-Assisted Multi-Modal Instruction-Tuning Dataset, Framework, and Benchmark](https://mlsanity.com/paper/2306.06687/) (2023)
- [CLIP2Point: Transfer CLIP to Point Cloud Classification with Image-Depth Pre-training](https://mlsanity.com/paper/2210.01055/) (2022)
- [ImVoxelNet: Image to Voxels Projection for Monocular and Multi-View General-Purpose 3D Object Detection](https://mlsanity.com/paper/2106.01178/) (2021)
- [EmbodiedScan: A Holistic Multi-Modal 3D Perception Suite Towards Embodied AI](https://mlsanity.com/paper/2312.16170/) (2023)
- [Bottom Up Top Down Detection Transformers for Language Grounding in Images and Point Clouds](https://mlsanity.com/paper/2112.08879/) (2021)
- [EDA: Explicit Text-Decoupling and Dense Alignment for 3D Visual Grounding](https://mlsanity.com/paper/2209.14941/) (2022)
- [LanguageRefer: Spatial-Language Model for 3D Visual Grounding](https://mlsanity.com/paper/2107.03438/) (2021)
- [Free-form Description Guided 3D Visual Graph Network for Object Grounding in Point Cloud](https://mlsanity.com/paper/2103.16381/) (2021)
- [CLIP goes 3D: Leveraging Prompt Tuning for Language Grounded 3D Recognition](https://mlsanity.com/paper/2303.11313/) (2023)
- [Visual Programming for Zero-shot Open-Vocabulary 3D Visual Grounding](https://mlsanity.com/paper/2311.15383/) (2023)
- [Look Around and Refer: 2D Synthetic Semantics Knowledge Distillation for 3D Visual Grounding](https://mlsanity.com/paper/2211.14241/) (2022)
- [3D-STMN: Dependency-Driven Superpoint-Text Matching Network for End-to-End 3D Referring Expression Segmentation](https://mlsanity.com/paper/2308.16632/) (2023)
- [MiKASA: Multi-Key-Anchor & Scene-Aware Transformer for 3D Visual Grounding](https://mlsanity.com/paper/2403.03077/) (2024)
- [Object2Scene: Putting Objects in Context for Open-Vocabulary 3D Detection](https://mlsanity.com/paper/2309.09456/) (2023)
- [EPCL: Frozen CLIP Transformer is An Efficient Point Cloud Encoder](https://mlsanity.com/paper/2212.04098/) (2022)

## Built on by (most cited first, 2 of 2)

- [3D-MRL: Nested Multimodal 3D Representations via Matryoshka Representation Learning](https://mlsanity.com/paper/2608.29285/) (2026)
- [Toward Unified Multimodal Representation Learning for Autonomous Driving](https://mlsanity.com/paper/2603.07874/) (2026)

## Abstract (arXiv)

3D visual grounding allows an embodied agent to understand visual information in real-world 3D environments based on human instructions, which is crucial for embodied intelligence. Existing 3D visual grounding methods typically rely on separate encoders for different modalities (e.g., RGB images, text, and 3D point clouds), resulting in large and complex models that are inefficient to train. While some approaches use pre-trained 2D multi-modal models like CLIP for 3D tasks, they still struggle with aligning point cloud data to 2D encoders. As a result, these methods continue to depend on 3D encoders for feature extraction, further increasing model complexity and training inefficiency. In this paper, we propose a unified 2D pre-trained multi-modal network to process all three modalities (RGB images, text, and point clouds), significantly simplifying the architecture. By leveraging a 2D CLIP bi-modal model with adapter-based fine-tuning, this framework effectively adapts to the tri-modal setting, improving both adaptability and performance across modalities. Our Geometric-Aware 2D-3D Feature Recovery and Fusion (GARF) module is designed to fuse geometric multi-scale features from point clouds and images. We then integrate textual features for final modality fusion and introduce a multi-modal decoder to facilitate deep cross-modal understanding. Together, our method achieves unified feature extraction and fusion across the three modalities, enabling an end-to-end 3D visual grounding model. Compared to the baseline, our method reduces the number of trainable parameters by approximately 58%, while achieving a 6.52% improvement in the 3D detection task and a 6.25% improvement in the 3D visual grounding task.