mlsanityData to 7 Oct 2026

Autoencoders as Cross-Modal Teachers: Can Pretrained 2D Image Transformers Help 3D Representation Learning?

arXiv 2212.08320 ↗First posted 16 Dec 2022cs.CVcited by 121

Not on the map. Why?

Abstract (arXiv)

The success of deep learning heavily relies on large-scale data with comprehensive labels, which is more expensive and time-consuming to fetch in 3D compared to 2D images or natural languages. This promotes the potential of utilizing models pretrained with data more than 3D as teachers for cross-modal knowledge transferring. In this paper, we revisit masked modeling in a unified fashion of knowledge distillation, and we show that foundational Transformers pretrained with 2D images or natural languages can help self-supervised 3D representation learning through training Autoencoders as Cross-Modal Teachers (ACT). The pretrained Transformers are transferred as cross-modal 3D teachers using discrete variational autoencoding self-supervision, during which the Transformers are frozen with prompt tuning for better knowledge inheritance. The latent features encoded by the 3D teachers are used as the target of masked point modeling, wherein the dark knowledge is distilled to the 3D Transformer students as foundational geometry understanding. Our ACT pretrained 3D learner achieves state-of-the-art generalization capacity across various downstream benchmarks, e.g., 88.21% overall accuracy on ScanObjectNN. Codes have been released at https://github.com/RunpeiDong/ACT.

Key facts. Cited by 121 papers in the map (data through 7 Oct 2026). Builds on 94 of its 122 references that are in the map. First posted on arXiv 16 Dec 2022.

Builds on

94 of 122 references.

References · 94

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  5. 2020Language Models are Few-Shot Learnerscited by 32,199
  6. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  7. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  8. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  9. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  10. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  11. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  12. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
38 more
  1. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  2. 2016PointNet: Deep Learning on Point Sets for 3D Classification and Segmentationcited by 5,223
  3. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
  4. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  5. 2017PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Spacecited by 4,384
  6. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  7. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  8. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  9. 2017ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenescited by 3,088
  10. 2015ShapeNet: An Information-Rich 3D Model Repositorycited by 2,880
  11. 2020Exploring Simple Siamese Representation Learningcited by 2,853
  12. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,360
  13. 2018Dynamic Graph CNN for Learning on Point Cloudscited by 2,220
  14. 2018Unsupervised Feature Learning via Non-Parametric Instance-level Discriminationcited by 1,838
  15. 2022Visual Prompt Tuningcited by 1,674
  16. 2014FitNets: Hints for Thin Deep Netscited by 1,619
  17. 2020Point Transformercited by 1,462
  18. 2021MLP-Mixer: An all-MLP Architecture for Visioncited by 1,441
  19. 2016Context Encoders: Feature Learning by Inpaintingcited by 1,424
  20. 2016Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzlescited by 1,379
  21. 2018Learning deep representations by mutual information estimation and maximizationcited by 1,265
  22. 2019KPConv: Flexible and Deformable Convolution for Point Cloudscited by 1,202
  23. 2019TinyBERT: Distilling BERT for Natural Language Understandingcited by 1,153
  24. 2016Colorful Image Colorizationcited by 1,113
  25. 2016Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfercited by 1,059
  26. 2016Deep Networks with Stochastic Depthcited by 973
  27. 2021iBOT: Image BERT Pre-Training with Online Tokenizercited by 891
  28. 2021Towards a Unified View of Parameter-Efficient Transfer Learningcited by 863
  29. 2018PointCNN: Convolution On X-Transformed Pointscited by 764
  30. 2020PCT: Point cloud transformercited by 730
  31. 2018PointConv: Deep Convolutional Networks on 3D Point Cloudscited by 656
  32. 2019Deep Hough Voting for 3D Object Detection in Point Cloudscited by 648
  33. 2021Point-BERT: Pre-training 3D Point Cloud Transformers with Masked Point Modelingcited by 647
  34. 2021Masked Feature Prediction for Self-Supervised Visual Pre-Trainingcited by 573
  35. 2019Revisiting Point Cloud Classification: A New Benchmark Dataset and Classification Model on Real-World Datacited by 532
  36. 2022Image as a Foreign Language: BEiT Pretraining for All Vision and Vision-Language Taskscited by 510
  37. 2020PointContrast: Unsupervised Pre-training for 3D Point Cloud Understandingcited by 502
  38. 2019Be Your Own Teacher: Improve the Performance of Convolutional Neural Networks via Self Distillationcited by 491

50 most cited of 94 All 94 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 32; 2024: 36; 2025: 35; 2026: 18.

121 papers.

Sort:

By area:

  1. 2023DreamLLM: Synergistic Multimodal Comprehension and Creationcited by 296
  2. 2025DreamVLA: A Vision-Language-Action Model Dreamed with Comprehensive World Knowledgecited by 203
  3. 2023ULIP-2: Towards Scalable Multimodal Pre-training for 3D Understandingcited by 199
  4. 2023Point-Bind & Point-LLM: Aligning Point Cloud with Multi-modality for 3D Understanding, Generation, and Instruction Followingcited by 192
  5. 2023Chat-Scene: Bridging 3D Scene and Large Language Models with Object Identifierscited by 170
  6. 2024PointMamba: A Simple State Space Model for Point Cloud Analysiscited by 166
  7. 2023Contrast with Reconstruct: Contrastive 3D Representation Learning Guided by Generative Pretrainingcited by 162
  8. 2023Meta-Transformer: A Unified Framework for Multimodal Learningcited by 136
  9. 2024ShapeLLM: Universal 3D Object Understanding for Embodied Interactioncited by 133
  10. 2024DreamBench++: A Human-Aligned Benchmark for Personalized Image Generationcited by 131
  11. 2023Chat-3D: Data-efficiently Tuning Large Language Model for Universal Dialogue of 3D Scenescited by 125
  12. 2023PointGPT: Auto-regressively Generative Pre-training from Point Cloudscited by 115
38 more
  1. 2023Swin3D: A Pretrained Transformer Backbone for 3D Indoor Scene Understandingcited by 111
  2. 2025OmniSpatial: Towards Comprehensive Spatial Reasoning Benchmark for Vision Language Modelscited by 105
  3. 2023CLIP-FO3D: Learning Free Open-world 3D Scene Representations from 2D Dense CLIPcited by 89
  4. 2025GeoVLA: Empowering 3D Representations in Vision-Language-Action Modelscited by 68
  5. 2023Instance-aware Dynamic Prompt Tuning for Pre-trained Point Cloud Modelscited by 67
  6. 2024Mamba3D: Enhancing Local Features for 3D Point Cloud Analysis via State Space Modelcited by 57
  7. 2024Dynamic Adapter Meets Prompt Tuning: Parameter-Efficient Transfer Learning for Point Cloud Analysiscited by 52
  8. 2025SoFar: Language-Grounded Orientation Bridges Spatial Reasoning and Object Manipulationcited by 46
  9. 2023Three Pillars improving Vision Foundation Model Distillation for Lidarcited by 43
  10. 2024Parameter-Efficient Fine-Tuning in Spectral Domain for Point Cloud Learningcited by 43
  11. 2023Point-PEFT: Parameter-Efficient Fine-Tuning for 3D Pre-trained Modelscited by 42
  12. 2023Towards Compact 3D Representations via Point Feature Enhancement Masked Autoencoderscited by 42
  13. 2023Visual Tuningcited by 34
  14. 2023Point2Vec for Self-Supervised Representation Learning on Point Cloudscited by 28
  15. 2023Bridging the Domain Gap: Self-Supervised 3D Scene Understanding with Foundation Modelscited by 26
  16. 2024SUGAR: Pre-training 3D Visual Representations for Roboticscited by 26
  17. 2023Take-A-Photo: 3D-to-2D Generative Pre-training of Point Cloud Modelscited by 25
  18. 2024Any2Point: Empowering Any-modality Large Models for Efficient 3D Understandingcited by 23
  19. 2024PCP-MAE: Learning to Predict Centers for Point Masked Autoencoderscited by 23
  20. 2023Self-supervised Learning for Pre-Training 3D Point Clouds: A Surveycited by 21
  21. 2024ShapeSplat: A Large-scale Dataset of Gaussian Splats and Their Self-Supervised Pretrainingcited by 21
  22. 2025DexVLG: Dexterous Vision-Language-Grasp Model at Scalecited by 19
  23. 2025Reasoning in Space via Grounding in the Worldcited by 18
  24. 2023Sculpting Holistic 3D Representation in Contrastive Language-Image-3D Pre-trainingcited by 17
  25. 2024Parameter-efficient Prompt Learning for 3D Point Cloud Understandingcited by 16
  26. 2024Bringing Masked Autoencoders Explicit Contrastive Properties for Point Cloud Self-Supervised Learningcited by 16
  27. 20233DAxiesPrompts: Unleashing the 3D Spatial Task Capabilities of GPT-4Vcited by 15
  28. 2024Positional Prompt Tuning for Efficient 3D Representation Learningcited by 15
  29. 2024RI-MAE: Rotation-Invariant Masked AutoEncoders for Self-Supervised Point Cloud Representation Learningcited by 15
  30. 2023Point-GCC: Universal Self-supervised 3D Scene Pre-training via Geometry-Color Contrastcited by 14
  31. 2023VPP: Efficient Conditional 3D Generation via Voxel-Point Progressive Representationcited by 14
  32. 20243D-JEPA: A Joint Embedding Predictive Architecture for 3D Self-Supervised Representation Learningcited by 14
  33. 2023Masked Modeling for Self-supervised Representation Learning on Vision and Beyondcited by 13
  34. 2024Adapter-X: A Novel General Parameter-Efficient Fine-Tuning Framework for Visioncited by 13
  35. 2025PointLoRA: Low-Rank Adaptation with Token Selection for Point Cloud Learningcited by 13
  36. 2025GAPrompt: Geometry-Aware Point Cloud Prompt for 3D Vision Modelcited by 13
  37. 2024LCM: Locally Constrained Compact Point Cloud Model for Masked Point Modelingcited by 12
  38. 2025PointDico: Contrastive 3D Representation Learning Guided by Diffusion Modelscited by 11

50 most cited of 121 All 121 (JSON)