mlsanityData to 7 Oct 2026

Multi-modal Relation Distillation for Unified 3D Representation Learning

arXiv 2407.14007 ↗First posted 19 Jul 2024cs.CV cs.AIcited by 5

Not on the map. Why?

Abstract (arXiv)

Recent advancements in multi-modal pre-training for 3D point clouds have demonstrated promising results by aligning heterogeneous features across 3D shapes and their corresponding 2D images and language descriptions. However, current straightforward solutions often overlook intricate structural relations among samples, potentially limiting the full capabilities of multi-modal learning. To address this issue, we introduce Multi-modal Relation Distillation (MRD), a tri-modal pre-training framework, which is designed to effectively distill reputable large Vision-Language Models (VLM) into 3D backbones. MRD aims to capture both intra-relations within each modality as well as cross-relations between different modalities and produce more discriminative 3D shape representations. Notably, MRD achieves significant improvements in downstream zero-shot classification tasks and cross-modality retrieval tasks, delivering new state-of-the-art performance.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Builds on 52 of its 61 references that are in the map. First posted on arXiv 19 Jul 2024.

Builds on

52 of 61 references.

References · 52

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  3. 2017PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Spacecited by 4,384
  4. 2015ShapeNet: An Information-Rich 3D Model Repositorycited by 2,880
  5. 2018DARTS: Differentiable Architecture Searchcited by 1,782
  6. 2022Objaverse: A Universe of Annotated 3D Objectscited by 1,522
  7. 2020Center-based 3D Object Detection and Trackingcited by 1,143
  8. 2019Relational Knowledge Distillationcited by 873
  9. 2021Point-BERT: Pre-training 3D Point Cloud Transformers with Masked Point Modelingcited by 647
  10. 2021FILIP: Fine-grained Interactive Language-Image Pre-Trainingcited by 626
  11. 2022Mind the Gap: Understanding the Modality Gap in Multi-modal Contrastive Representation Learningcited by 553
  12. 2019Revisiting Point Cloud Classification: A New Benchmark Dataset and Classification Model on Real-World Datacited by 532
38 more
  1. 2020PointContrast: Unsupervised Pre-training for 3D Point Cloud Understandingcited by 502
  2. 2021PointCLIP: Point Cloud Understanding by CLIPcited by 461
  3. 2022Masked Autoencoders for Point Cloud Self-supervised Learningcited by 456
  4. 2022PointNeXt: Revisiting PointNet++ with Improved Training and Scaling Strategiescited by 432
  5. 2021ABO: Dataset and Benchmarks for Real-World 3D Object Understandingcited by 360
  6. 2021An End-to-End Transformer Model for 3D Object Detectioncited by 329
  7. 20203D-FUTURE: 3D Furniture shape with TextUREcited by 306
  8. 2022Point-M2AE: Multi-scale Masked Autoencoders for Hierarchical Point Cloud Pre-trainingcited by 263
  9. 2022ULIP: Learning a Unified Representation of Language, Images, and Point Clouds for 3D Understandingcited by 262
  10. 2023Uni3D: Exploring Unified 3D Representation at Scalecited by 231
  11. 2022DeepFusion: Lidar-Camera Deep Fusion for Multi-Modal 3D Object Detectioncited by 224
  12. 2021Group-Free 3D Object Detection via Transformerscited by 215
  13. 2023Scalable 3D Captioning with Pretrained Modelscited by 201
  14. 2023ULIP-2: Towards Scalable Multimodal Pre-training for 3D Understandingcited by 199
  15. 2022SoftGroup for 3D Instance Segmentation on Point Cloudscited by 197
  16. 2020Unsupervised Point Cloud Pre-Training via Occlusion Completioncited by 186
  17. 2022PointCLIP V2: Prompting CLIP and GPT for Powerful 3D Open-world Learningcited by 186
  18. 2023OpenShape: Scaling Up 3D Shape Representation Towards Open-World Understandingcited by 183
  19. 2021Spatio-temporal Self-Supervised Representation Learning for 3D Point Cloudscited by 180
  20. 2019PU-GAN: a Point Cloud Upsampling Adversarial Networkcited by 162
  21. 2023Contrast with Reconstruct: Contrastive 3D Representation Learning Guided by Generative Pretrainingcited by 162
  22. 2022CLIP2Point: Transfer CLIP to Point Cloud Classification with Image-Depth Pre-trainingcited by 159
  23. 2022Learning 3D Representations from 2D Pre-trained Models via Image-to-Point Masked Autoencoderscited by 143
  24. 2022CyCLIP: Cyclic Contrastive Language-Image Pretrainingcited by 127
  25. 2021Multimodal Contrastive Training for Visual Representation Learningcited by 122
  26. 2022Autoencoders as Cross-Modal Teachers: Can Pretrained 2D Image Transformers Help 3D Representation Learning?cited by 121
  27. 2023PointGPT: Auto-regressively Generative Pre-training from Point Cloudscited by 115
  28. 2022PyramidCLIP: Hierarchical Feature Alignment for Vision-language Model Pretrainingcited by 104
  29. 2023CLIP goes 3D: Leveraging Prompt Tuning for Language Grounded 3D Recognitioncited by 77
  30. 2020Global-Local Bidirectional Reasoning for Unsupervised Representation Learning of 3D Point Cloudscited by 75
  31. 2020Self-supervised Learning of Point Clouds via Orientation Estimationcited by 72
  32. 2023SoftCLIP: Softer Cross-modal Alignment Makes CLIP Strongercited by 56
  33. 2022Robust Cross-Modal Representation Learning with Progressive Self-Distillationcited by 55
  34. 2023PiMAE: Point Cloud and Image Interactive Masked Autoencoders for 3D Object Detectioncited by 53
  35. 2023Joint-MAE: 2D-3D Joint Masked Autoencoders for 3D Point Cloud Pre-trainingcited by 52
  36. 2022Implicit Autoencoder for Point-Cloud Self-Supervised Representation Learningcited by 46
  37. 2022Ranking Info Noise Contrastive Estimation: Boosting Contrastive Learning via Ranked Positivescited by 37
  38. 2023Connecting Multi-modal Contrastive Representationscited by 36

50 most cited of 52 All 52 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 2; 2026: 2.

5 papers.

  1. 2024Positional Prompt Tuning for Efficient 3D Representation Learningcited by 15
  2. 2026Tango3D: Towards Alignment for Global and Local 2D-3D Correspondencecited by 2
  3. 20263D-MRL: Nested Multimodal 3D Representations via Matryoshka Representation Learningcited by 1
  4. 2025SAMURAI: Shape-Aware Multimodal Retrieval for 3D Object Identificationcited by 0
  5. 2025HumanCrafter: Synergizing Generalizable Human Reconstruction and Semantic 3D Segmentationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Beyond First Impressions: Integrating Joint Multi-modal Cues for Comprehensive 3D Representation
  2. 2024Learning Modality-agnostic Representation for Semantic Segmentation from Any Modalities
  3. 2025Collaborative Multi-Modal Coding for High-Quality 3D Generation
  4. 2024Cross-Modal Self-Training: Aligning Images and Pointclouds to Learn Classification without Labels
  5. 20253D-Aware Vision-Language Models Fine-Tuning with Geometric Distillation
  6. 2025Inst3D-LMM: Instance-Aware 3D Scene Understanding with Multi-modal Instruction Tuning
  7. 20243D Scene Graph Guided Vision-Language Pre-training
  8. 2023Point-Bind & Point-LLM: Aligning Point Cloud with Multi-modality for 3D Understanding, Generation, and Instruction Following