# TAMM: TriAdapter Multi-Modal Learning for 3D Shape Understanding

arXiv: https://arxiv.org/abs/2402.18490 · HTML: https://mlsanity.com/paper/2402.18490/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- Builds on 57 of its 65 references that are in the map.
- First posted on arXiv 28 Feb 2024.

## Builds on (most cited first, 30 of 57)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation](https://mlsanity.com/paper/1612.00593/) (2016)
- [PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space](https://mlsanity.com/paper/1706.02413/) (2017)
- [BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation](https://mlsanity.com/paper/2201.12086/) (2022)
- [Parameter-Efficient Transfer Learning for NLP](https://mlsanity.com/paper/1902.00751/) (2019)
- [LAION-5B: An open large-scale dataset for training next generation image-text models](https://mlsanity.com/paper/2210.08402/) (2022)
- [Gaussian Error Linear Units (GELUs)](https://mlsanity.com/paper/1606.08415/) (2016)
- [ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenes](https://mlsanity.com/paper/1702.04405/) (2017)
- [ShapeNet: An Information-Rich 3D Model Repository](https://mlsanity.com/paper/1512.03012/) (2015)
- [Objaverse: A Universe of Annotated 3D Objects](https://mlsanity.com/paper/2212.08051/) (2022)
- [LVIS: A Dataset for Large Vocabulary Instance Segmentation](https://mlsanity.com/paper/1908.03195/) (2019)
- [4D Spatio-Temporal ConvNets: Minkowski Convolutional Neural Networks](https://mlsanity.com/paper/1904.08755/) (2019)
- [Reproducible scaling laws for contrastive language-image learning](https://mlsanity.com/paper/2212.07143/) (2022)
- [Grounded Language-Image Pre-training](https://mlsanity.com/paper/2112.03857/) (2021)
- [CLIP-Adapter: Better Vision-Language Models with Feature Adapters](https://mlsanity.com/paper/2110.04544/) (2021)
- [PV-RCNN: Point-Voxel Feature Set Abstraction for 3D Object Detection](https://mlsanity.com/paper/1912.13192/) (2019)
- [Point-BERT: Pre-training 3D Point Cloud Transformers with Masked Point Modeling](https://mlsanity.com/paper/2111.14819/) (2021)
- [RegionCLIP: Region-based Language-Image Pretraining](https://mlsanity.com/paper/2112.09106/) (2021)
- [Detecting Twenty-thousand Classes using Image-level Supervision](https://mlsanity.com/paper/2201.02605/) (2022)
- [Hypersim: A Photorealistic Synthetic Dataset for Holistic Indoor Scene Understanding](https://mlsanity.com/paper/2011.02523/) (2020)
- [OpenScene: 3D Scene Understanding with Open Vocabularies](https://mlsanity.com/paper/2211.15654/) (2022)
- [Revisiting Point Cloud Classification: A New Benchmark Dataset and Classification Model on Real-World Data](https://mlsanity.com/paper/1908.04616/) (2019)

## Built on by (most cited first, 27 of 27)

- [Rethinking Vision-Language Model in Face Forensics: Multi-Modal Interpretable Forged Face Detector](https://mlsanity.com/paper/2503.20188/) (2025)
- [Language-guided Hierarchical Fine-grained Image Forgery Detection and Localization](https://mlsanity.com/paper/2410.23556/) (2024)
- [Safety Mirage: How Spurious Correlations Undermine VLM Safety Fine-Tuning and Can Be Mitigated by Machine Unlearning](https://mlsanity.com/paper/2503.11832/) (2025)
- [Duoduo CLIP: Efficient 3D Understanding with Multi-View Images](https://mlsanity.com/paper/2406.11579/) (2024)
- [Unlearning Isn't Invisible: Detecting Unlearning Traces in LLMs from Model Outputs](https://mlsanity.com/paper/2506.14003/) (2025)
- [DreamView: Injecting View-specific Text Guidance into Text-to-3D Generation](https://mlsanity.com/paper/2404.06119/) (2024)
- [OpenDlign: Open-World Point Cloud Understanding with Depth-Aligned Images](https://mlsanity.com/paper/2404.16538/) (2024)
- [Swiss Army Knife: Synergizing Biases in Knowledge from Vision Foundation Models for Multi-Task Learning](https://mlsanity.com/paper/2410.14633/) (2024)
- [Efficient and Effective Adaptation of Multimodal Foundation Models in Sequential Recommendation](https://mlsanity.com/paper/2411.02992/) (2024)
- [UniGS: Unified Language-Image-3D Pretraining with Gaussian Splatting](https://mlsanity.com/paper/2502.17860/) (2025)
- [FusionAgent: A Multimodal Agent with Dynamic Model Selection for Human Recognition](https://mlsanity.com/paper/2603.26908/) (2026)
- [Towards Intrinsic-Aware Monocular 3D Object Detection](https://mlsanity.com/paper/2603.27059/) (2026)
- [EmoTaG: Emotion-Aware Talking Head Synthesis on Gaussian Splatting with Few-Shot Personalization](https://mlsanity.com/paper/2603.21332/) (2026)
- [Accelerating Non-Maximum Suppression: A Graph Theory Perspective](https://mlsanity.com/paper/2409.20520/) (2024)
- [Occlusion-aware Text-Image-Point Cloud Pretraining for Open-World 3D Object Recognition](https://mlsanity.com/paper/2502.10674/) (2025)
- [Point Cloud as a Foreign Language for Multi-modal Large Language Model](https://mlsanity.com/paper/2603.09173/) (2026)
- [TIGaussian: Disentangle Gaussians for Spatial-Awared Text-Image-3D Alignment](https://mlsanity.com/paper/2601.19247/) (2026)
- [PartRAG: Retrieval-Augmented Part-Level 3D Generation and Editing](https://mlsanity.com/paper/2602.17033/) (2026)
- [3D-MRL: Nested Multimodal 3D Representations via Matryoshka Representation Learning](https://mlsanity.com/paper/2608.29285/) (2026)
- [SCENEFORGE: Enhancing 3D-text alignment with Structured Scene Compositions](https://mlsanity.com/paper/2509.15693/) (2025)
- [RI-Mamba: Rotation-Invariant Mamba for Robust Text-to-Shape Retrieval](https://mlsanity.com/paper/2602.11673/) (2026)
- [Pointy - A Lightweight Transformer for Point Cloud Foundation Models](https://mlsanity.com/paper/2603.10963/) (2026)
- [Multi-modal Test-time Adaptation via Adaptive Probabilistic Gaussian Calibration](https://mlsanity.com/paper/2604.19093/) (2026)
- [HOLA: Holistic Multi-Modal Alignment for Open-Set 3D Recognition](https://mlsanity.com/paper/2606.01334/) (2026)
- [Who Built This Model? Tracing LLM Lineage via Spectral Fingerprints in Weight Space](https://mlsanity.com/paper/2608.07786/) (2026)
- [An overview of 3D Vision-Language Models](https://mlsanity.com/paper/2609.05583/) (2026)
- [Revisiting Triplet Contrastive Learning for 3D Shape Understanding](https://mlsanity.com/paper/2610.19519/) (2026)

## Abstract (arXiv)

The limited scale of current 3D shape datasets hinders the advancements in 3D shape understanding, and motivates multi-modal learning approaches which transfer learned knowledge from data-abundant 2D image and language modalities to 3D shapes. However, even though the image and language representations have been aligned by cross-modal models like CLIP, we find that the image modality fails to contribute as much as the language in existing multi-modal 3D representation learning methods. This is attributed to the domain shift in the 2D images and the distinct focus of each modality. To more effectively leverage both modalities in the pre-training, we introduce TriAdapter Multi-Modal Learning (TAMM) -- a novel two-stage learning approach based on three synergistic adapters. First, our CLIP Image Adapter mitigates the domain gap between 3D-rendered images and natural images, by adapting the visual representations of CLIP for synthetic image-text pairs. Subsequently, our Dual Adapters decouple the 3D shape representation space into two complementary sub-spaces: one focusing on visual attributes and the other for semantic understanding, which ensure a more comprehensive and effective multi-modal pre-training. Extensive experiments demonstrate that TAMM consistently enhances 3D representations for a wide range of 3D encoder architectures, pre-training datasets, and downstream tasks. Notably, we boost the zero-shot classification accuracy on Objaverse-LVIS from 46.8% to 50.7%, and improve the 5-way 10-shot linear probing classification accuracy on ModelNet40 from 96.1% to 99.0%. Project page: https://alanzhangcs.github.io/tamm-page.