# Multi-modal Relation Distillation for Unified 3D Representation Learning

arXiv: https://arxiv.org/abs/2407.14007 · HTML: https://mlsanity.com/paper/2407.14007/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 52 of its 61 references that are in the map.
- First posted on arXiv 19 Jul 2024.

## Builds on (most cited first, 30 of 52)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space](https://mlsanity.com/paper/1706.02413/) (2017)
- [ShapeNet: An Information-Rich 3D Model Repository](https://mlsanity.com/paper/1512.03012/) (2015)
- [DARTS: Differentiable Architecture Search](https://mlsanity.com/paper/1806.09055/) (2018)
- [Objaverse: A Universe of Annotated 3D Objects](https://mlsanity.com/paper/2212.08051/) (2022)
- [Center-based 3D Object Detection and Tracking](https://mlsanity.com/paper/2006.11275/) (2020)
- [Relational Knowledge Distillation](https://mlsanity.com/paper/1904.05068/) (2019)
- [Point-BERT: Pre-training 3D Point Cloud Transformers with Masked Point Modeling](https://mlsanity.com/paper/2111.14819/) (2021)
- [FILIP: Fine-grained Interactive Language-Image Pre-Training](https://mlsanity.com/paper/2111.07783/) (2021)
- [Mind the Gap: Understanding the Modality Gap in Multi-modal Contrastive Representation Learning](https://mlsanity.com/paper/2203.02053/) (2022)
- [Revisiting Point Cloud Classification: A New Benchmark Dataset and Classification Model on Real-World Data](https://mlsanity.com/paper/1908.04616/) (2019)
- [PointContrast: Unsupervised Pre-training for 3D Point Cloud Understanding](https://mlsanity.com/paper/2007.10985/) (2020)
- [PointCLIP: Point Cloud Understanding by CLIP](https://mlsanity.com/paper/2112.02413/) (2021)
- [Masked Autoencoders for Point Cloud Self-supervised Learning](https://mlsanity.com/paper/2203.06604/) (2022)
- [PointNeXt: Revisiting PointNet++ with Improved Training and Scaling Strategies](https://mlsanity.com/paper/2206.04670/) (2022)
- [ABO: Dataset and Benchmarks for Real-World 3D Object Understanding](https://mlsanity.com/paper/2110.06199/) (2021)
- [An End-to-End Transformer Model for 3D Object Detection](https://mlsanity.com/paper/2109.08141/) (2021)
- [3D-FUTURE: 3D Furniture shape with TextURE](https://mlsanity.com/paper/2009.09633/) (2020)
- [ULIP: Learning a Unified Representation of Language, Images, and Point Clouds for 3D Understanding](https://mlsanity.com/paper/2212.05171/) (2022)
- [Point-M2AE: Multi-scale Masked Autoencoders for Hierarchical Point Cloud Pre-training](https://mlsanity.com/paper/2205.14401/) (2022)
- [Uni3D: Exploring Unified 3D Representation at Scale](https://mlsanity.com/paper/2310.06773/) (2023)
- [DeepFusion: Lidar-Camera Deep Fusion for Multi-Modal 3D Object Detection](https://mlsanity.com/paper/2203.08195/) (2022)
- [Group-Free 3D Object Detection via Transformers](https://mlsanity.com/paper/2104.00678/) (2021)
- [Scalable 3D Captioning with Pretrained Models](https://mlsanity.com/paper/2306.07279/) (2023)
- [ULIP-2: Towards Scalable Multimodal Pre-training for 3D Understanding](https://mlsanity.com/paper/2305.08275/) (2023)
- [SoftGroup for 3D Instance Segmentation on Point Clouds](https://mlsanity.com/paper/2203.01509/) (2022)
- [PointCLIP V2: Prompting CLIP and GPT for Powerful 3D Open-world Learning](https://mlsanity.com/paper/2211.11682/) (2022)
- [Unsupervised Point Cloud Pre-Training via Occlusion Completion](https://mlsanity.com/paper/2010.01089/) (2020)
- [OpenShape: Scaling Up 3D Shape Representation Towards Open-World Understanding](https://mlsanity.com/paper/2305.10764/) (2023)

## Built on by (most cited first, 6 of 6)

- [Positional Prompt Tuning for Efficient 3D Representation Learning](https://mlsanity.com/paper/2408.11567/) (2024)
- [Tango3D: Towards Alignment for Global and Local 2D-3D Correspondence](https://mlsanity.com/paper/2605.19727/) (2026)
- [3D-MRL: Nested Multimodal 3D Representations via Matryoshka Representation Learning](https://mlsanity.com/paper/2608.29285/) (2026)
- [SAMURAI: Shape-Aware Multimodal Retrieval for 3D Object Identification](https://mlsanity.com/paper/2506.21056/) (2025)
- [HumanCrafter: Synergizing Generalizable Human Reconstruction and Semantic 3D Segmentation](https://mlsanity.com/paper/2511.00468/) (2025)
- [Revisiting Triplet Contrastive Learning for 3D Shape Understanding](https://mlsanity.com/paper/2610.19519/) (2026)

## Abstract (arXiv)

Recent advancements in multi-modal pre-training for 3D point clouds have demonstrated promising results by aligning heterogeneous features across 3D shapes and their corresponding 2D images and language descriptions. However, current straightforward solutions often overlook intricate structural relations among samples, potentially limiting the full capabilities of multi-modal learning. To address this issue, we introduce Multi-modal Relation Distillation (MRD), a tri-modal pre-training framework, which is designed to effectively distill reputable large Vision-Language Models (VLM) into 3D backbones. MRD aims to capture both intra-relations within each modality as well as cross-relations between different modalities and produce more discriminative 3D shape representations. Notably, MRD achieves significant improvements in downstream zero-shot classification tasks and cross-modality retrieval tasks, delivering new state-of-the-art performance.