mlsanityData to 7 Oct 2026

Joint-MAE: 2D-3D Joint Masked Autoencoders for 3D Point Cloud Pre-training

arXiv 2302.14007 ↗First posted 27 Feb 2023cs.CVcited by 52

Not on the map. Why?

Abstract (arXiv)

Masked Autoencoders (MAE) have shown promising performance in self-supervised learning for both 2D and 3D computer vision. However, existing MAE-style methods can only learn from the data of a single modality, i.e., either images or point clouds, which neglect the implicit semantic and geometric correlation between 2D and 3D. In this paper, we explore how the 2D modality can benefit 3D masked autoencoding, and propose Joint-MAE, a 2D-3D joint MAE framework for self-supervised 3D point cloud pre-training. Joint-MAE randomly masks an input 3D point cloud and its projected 2D images, and then reconstructs the masked information of the two modalities. For better cross-modal interaction, we construct our JointMAE by two hierarchical 2D-3D embedding modules, a joint encoder, and a joint decoder with modal-shared and model-specific decoders. On top of this, we further introduce two cross-modal strategies to boost the 3D representation learning, which are local-aligned attention mechanisms for 2D-3D semantic cues, and a cross-reconstruction loss for 2D-3D geometric constraints. By our pre-training paradigm, Joint-MAE achieves superior performance on multiple downstream tasks, e.g., 92.4% accuracy for linear SVM on ModelNet40 and 86.07% accuracy on the hardest split of ScanObjectNN.

Key facts. Cited by 52 papers in the map (data through 7 Oct 2026). Builds on 54 of its 79 references that are in the map. First posted on arXiv 27 Feb 2023.

Builds on

54 of 79 references.

References · 54

  1. 2017Attention Is All You Needcited by 59,932
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  4. 2016PointNet: Deep Learning on Point Sets for 3D Classification and Segmentationcited by 5,223
  5. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
  6. 2017PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Spacecited by 4,384
  7. 2015ShapeNet: An Information-Rich 3D Model Repositorycited by 2,880
  8. 2018Dynamic Graph CNN for Learning on Point Cloudscited by 2,220
  9. 2020Point Transformercited by 1,462
  10. 2019KPConv: Flexible and Deformable Convolution for Point Cloudscited by 1,202
  11. 2021SimMIM: A Simple Framework for Masked Image Modelingcited by 1,147
  12. 2021CLIP-Adapter: Better Vision-Language Models with Feature Adapterscited by 1,021
38 more
  1. 2016A Point Set Generation Network for 3D Object Reconstruction from a Single Imagecited by 972
  2. 2018PointCNN: Convolution On X-Transformed Pointscited by 764
  3. 2020PCT: Point cloud transformercited by 730
  4. 2022Point-E: A System for Generating 3D Point Clouds from Complex Promptscited by 694
  5. 2021Point-BERT: Pre-training 3D Point Cloud Transformers with Masked Point Modelingcited by 647
  6. 2016Learning a Probabilistic Latent Space of Object Shapes via 3D Generative-Adversarial Modelingcited by 558
  7. 2019Revisiting Point Cloud Classification: A New Benchmark Dataset and Classification Model on Real-World Datacited by 532
  8. 2020PointContrast: Unsupervised Pre-training for 3D Point Cloud Understandingcited by 502
  9. 2021PointCLIP: Point Cloud Understanding by CLIPcited by 461
  10. 2022Masked Autoencoders for Point Cloud Self-supervised Learningcited by 456
  11. 2017Look, Listen and Learncited by 452
  12. 2019ScanRefer: 3D Object Localization in RGB-D Scans using Natural Languagecited by 446
  13. 2021Tip-Adapter: Training-free CLIP-Adapter for Better Vision-Language Modelingcited by 429
  14. 2022Tip-Adapter: Training-free Adaption of CLIP for Few-shot Classificationcited by 355
  15. 2019Relation-Shape Convolutional Neural Network for Point Cloud Analysiscited by 346
  16. 2020VirTex: Learning Visual Representations from Textual Annotationscited by 338
  17. 2022Point-M2AE: Multi-scale Masked Autoencoders for Hierarchical Point Cloud Pre-trainingcited by 263
  18. 2021Self-Supervised Pretraining of 3D Features on any Point-Cloudcited by 217
  19. 2018SO-Net: Self-Organizing Network for Point Cloud Analysiscited by 210
  20. 2022CrossPoint: Self-Supervised Cross-Modal Contrastive Learning for 3D Point Cloud Understandingcited by 198
  21. 2020Unsupervised Point Cloud Pre-Training via Occlusion Completioncited by 186
  22. 2022Frozen CLIP Models are Efficient Video Learnerscited by 181
  23. 2016A Theory of Generative ConvNetcited by 174
  24. 2022Masked Discrimination for Self-Supervised Learning on Point Cloudscited by 155
  25. 2021Revisiting Point Cloud Shape Classification with a Simple and Effective Baselinecited by 145
  26. 2022Learning 3D Representations from 2D Pre-trained Models via Image-to-Point Masked Autoencoderscited by 143
  27. 2021InstanceRefer: Cooperative Holistic Understanding for Visual Grounding on Point Clouds through Instance Multi-level Contextual Referringcited by 140
  28. 2022ConvMAE: Masked Convolution Meets Masked Autoencoderscited by 105
  29. 2022CALIP: Zero-Shot Enhancement of CLIP with Parameter-free Attentioncited by 103
  30. 2018Multiresolution Tree Networks for 3D Point Cloud Processingcited by 91
  31. 2020Global-Local Bidirectional Reasoning for Unsupervised Representation Learning of 3D Point Cloudscited by 75
  32. 2020Self-supervised Learning of Point Clouds via Orientation Estimationcited by 72
  33. 2022Can Language Understand Depth?cited by 66
  34. 2020Generative PointNet: Deep Energy-Based Learning on Unordered Point Sets for 3D Generation, Reconstruction and Classificationcited by 57
  35. 2018Learning Descriptor Networks for 3D Shape Synthesis and Analysiscited by 56
  36. 2018View Inter-Prediction GAN: Unsupervised Representation Learning for 3D Shapes by Learning Global Shape Memories to Support Local View Predictionscited by 43
  37. 2021VT-CLIP: Enhancing Vision-Language Models with Visual-guided Textscited by 42
  38. 2020Generative VoxelNet: Learning Energy-Based Models for 3D Shape Synthesis and Analysiscited by 26

50 most cited of 54 All 54 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 17; 2024: 18; 2025: 10; 2026: 7.

52 papers.

Sort:

By area:

  1. 2023Point-Bind & Point-LLM: Aligning Point Cloud with Multi-modality for 3D Understanding, Generation, and Instruction Followingcited by 192
  2. 2024PointMamba: A Simple State Space Model for Point Cloud Analysiscited by 166
  3. 2023Prompt, Generate, then Cache: Cascade of Foundation Models makes Strong Few-shot Learnerscited by 152
  4. 2023Not All Features Matter: Enhancing Few-shot CLIP with Adaptive Prior Refinementcited by 84
  5. 2023ViewRefer: Grasp the Multi-view Knowledge for 3D Visual Grounding with GPT and Prototype Guidancecited by 70
  6. 2023Point Cloud Pre-training with Diffusion Modelscited by 47
  7. 2023Traj-MAE: Masked Autoencoders for Trajectory Predictioncited by 42
  8. 2023Point-PEFT: Parameter-Efficient Fine-Tuning for 3D Pre-trained Modelscited by 42
  9. 2023Towards Compact 3D Representations via Point Feature Enhancement Masked Autoencoderscited by 42
  10. 2023Parameter is Not All You Need: Starting from Non-Parametric Networks for 3D Point Cloud Analysiscited by 34
  11. 2023Continual-MAE: Adaptive Distribution Masked Autoencoders for Continual Test-Time Adaptationcited by 30
  12. 2024Masked Image Modeling: A Surveycited by 28
38 more
  1. 2023Pedestrian Attribute Recognition via CLIP based Prompt Vision-Language Fusioncited by 24
  2. 2024Any2Point: Empowering Any-modality Large Models for Efficient 3D Understandingcited by 23
  3. 2024PCP-MAE: Learning to Predict Centers for Point Masked Autoencoderscited by 23
  4. 2024No Time to Train: Empowering Non-Parametric Networks for Few-shot 3D Scene Segmentationcited by 18
  5. 2024SAM2Point: Segment Any 3D as Videos in Zero-shot and Promptable Mannerscited by 18
  6. 2024Bringing Masked Autoencoders Explicit Contrastive Properties for Point Cloud Self-Supervised Learningcited by 16
  7. 20233DAxiesPrompts: Unleashing the 3D Spatial Task Capabilities of GPT-4Vcited by 15
  8. 2023Mimic before Reconstruct: Enhancing Masked Autoencoders with Feature Mimickingcited by 12
  9. 2024LCM: Locally Constrained Compact Point Cloud Model for Masked Point Modelingcited by 12
  10. 2023MV-CLIP: Multi-View CLIP for Zero-shot 3D Shape Recognitioncited by 11
  11. 2025PointDico: Contrastive 3D Representation Learning Guided by Diffusion Modelscited by 11
  12. 2025PointSea: Point Cloud Completion via Self-structure Augmentationcited by 9
  13. 2025Towards More Diverse and Challenging Pre-training for Point Cloud Learning: Self-Supervised Cross Reconstruction with Decoupled Viewscited by 7
  14. 2026Mantis: Mamba-native Tuning is Efficient for 3D Point Cloud Foundation Modelscited by 6
  15. 2024Multi-modal Relation Distillation for Unified 3D Representation Learningcited by 5
  16. 2026Tri-Efficient Transfer Learning for Point Cloud Videoscited by 5
  17. 2024Adapt PointFormer: 3D Point Cloud Analysis via Adapting 2D Visual Transformerscited by 4
  18. 2025Harnessing Text-to-Image Diffusion Models for Point Cloud Self-Supervised Learningcited by 3
  19. 2023Point Cloud Self-supervised Learning via 3D to Multi-view Masked Learnercited by 2
  20. 2024Pre-training Point Cloud Compact Model with Partial-aware Reconstructioncited by 2
  21. 2024InScope: A New Real-world 3D Infrastructure-side Collaborative Perception Dataset for Open Traffic Scenarioscited by 2
  22. 2025MambaTron: Efficient Cross-Modal Point Cloud Enhancement using Aggregate Selective State Space Modelingcited by 2
  23. 2025Masked Clustering Prediction for Unsupervised Point Cloud Pre-trainingcited by 2
  24. 2023Cross-BERT for Point Cloud Pretrainingcited by 1
  25. 2024MM-Mixing: Multi-Modal Mixing Alignment for 3D Understandingcited by 1
  26. 2024Semantic-Free Procedural 3D Shapes Are Surprisingly Good Teacherscited by 1
  27. 2025PointLAMA: Latent Attention meets Mamba for Efficient Point Cloud Pretrainingcited by 1
  28. 2025DAP-MAE: Domain-Adaptive Point Cloud Masked Autoencoder for Effective Cross-Domain Learningcited by 1
  29. 2026PointAlign: Feature-Level Alignment Regularization for 3D Vision-Language Modelscited by 1
  30. 2026Rethinking Point Clouds as Sequences: A Causal Next-Token Predictive Learning Frameworkcited by 1
  31. 2026From Extrinsic to Intrinsic: Geodesic-Guided Representation Learning for 3D Geometric Datacited by 1
  32. 2023Instance-aware 3D Semantic Segmentation powered by Shape Generators and Classifierscited by 0
  33. 2024Masked Generative Extractor for Synergistic Representation and 3D Generation of Point Cloudscited by 0
  34. 2024P3P: Pseudo-3D Pre-training for Scaling 3D Voxel-based Masked Autoencoderscited by 0
  35. 2024PMT-MAE: Dual-Branch Self-Supervised Learning with Distillation for Efficient Point Cloud Classificationcited by 0
  36. 2024PointCG: Self-supervised Point Cloud Learning via Joint Completion and Generationcited by 0
  37. 2025Adaptive Point-Prompt Tuning: Fine-Tuning Heterogeneous Foundation Models for 3D Point Cloud Analysiscited by 0
  38. 2025Dual-Branch Center-Surrounding Contrast: Rethinking Contrastive Learning for 3D Point Cloudscited by 0

50 most cited of 52 All 52 (JSON)