mlsanityData to 14 Oct 2026

Asymmetric Dual Self-Distillation for 3D Self-Supervised Representation Learning

arXiv 2506.21724 ↗First posted 26 Jun 2025cs.CVcited by 0

On the map: Knowledge distillation → Detection and dense prediction distillation

Abstract (arXiv)

Learning semantically meaningful representations from unstructured 3D point clouds remains a central challenge in computer vision, especially in the absence of large-scale labeled datasets. While masked point modeling (MPM) is widely used in self-supervised 3D learning, its reconstruction-based objective can limit its ability to capture high-level semantics. We propose AsymDSD, an Asymmetric Dual Self-Distillation framework that unifies masked modeling and invariance learning through prediction in the latent space rather than the input space. AsymDSD builds on a joint embedding architecture and introduces several key design choices: an efficient asymmetric setup, disabling attention between masked queries to prevent shape leakage, multi-mask sampling, and a point cloud adaptation of multi-crop. AsymDSD achieves state-of-the-art results on ScanObjectNN (90.53%) and further improves to 93.72% when pretrained on 930k shapes, surpassing prior methods.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Detection and dense prediction distillation, one of 12 directions in Knowledge distillation. Builds on 82 of its 95 references that are in the map. First posted on arXiv 26 Jun 2025.

Builds on

82 of 95 references.

Closest ancestors · 2

Same direction, newest first

  1. 2022Autoencoders as Cross-Modal Teachers: Can Pretrained 2D Image Transformers Help 3D Representation Learning?16 Dec
  2. 2022Self-Distillation for Unsupervised 3D Domain Adaptation15 Oct

Other references · 80

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  4. 2020Language Models are Few-Shot Learnerscited by 32,419
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  6. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  7. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  8. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,974
  9. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  10. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,513
  11. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
  12. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,968
38 more
  1. 2020Scaling Laws for Neural Language Modelscited by 6,586
  2. 2016PointNet: Deep Learning on Point Sets for 3D Classification and Segmentationcited by 5,248
  3. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,750
  4. 2020Training data-efficient image transformers & distillation through attentioncited by 4,586
  5. 2017PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Spacecited by 4,401
  6. 2021Zero-Shot Text-to-Image Generationcited by 4,288
  7. 2016Gaussian Error Linear Units (GELUs)cited by 3,502
  8. 2017ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenescited by 3,117
  9. 2015Fast R-CNNcited by 2,933
  10. 2015ShapeNet: An Information-Rich 3D Model Repositorycited by 2,897
  11. 2020Exploring Simple Siamese Representation Learningcited by 2,866
  12. 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,857
  13. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,375
  14. 2021Barlow Twins: Self-Supervised Learning via Redundancy Reductioncited by 1,838
  15. 2022Objaverse: A Universe of Annotated 3D Objectscited by 1,540
  16. 2019Root Mean Square Layer Normalizationcited by 1,480
  17. 2019LVIS: A Dataset for Large Vocabulary Instance Segmentationcited by 1,217
  18. 2017Mixed Precision Trainingcited by 1,156
  19. 2021SimMIM: A Simple Framework for Masked Image Modelingcited by 1,153
  20. 2021VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learningcited by 1,057
  21. 2023Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecturecited by 1,039
  22. 2016Deep Networks with Stochastic Depthcited by 979
  23. 2021iBOT: Image BERT Pre-Training with Online Tokenizercited by 903
  24. 2023Objaverse-XL: A Universe of 10M+ 3D Objectscited by 774
  25. 2022data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Languagecited by 727
  26. 2021Point-BERT: Pre-training 3D Point Cloud Transformers with Masked Point Modelingcited by 653
  27. 2022Google Scanned Objects: A High-Quality Dataset of 3D Scanned Household Itemscited by 612
  28. 2023Point Transformer V3: Simpler, Faster, Strongercited by 589
  29. 2019Revisiting Point Cloud Classification: A New Benchmark Dataset and Classification Model on Real-World Datacited by 532
  30. 2020PointContrast: Unsupervised Pre-training for 3D Point Cloud Understandingcited by 504
  31. 2022Masked Autoencoders for Point Cloud Self-supervised Learningcited by 459
  32. 2022PointNeXt: Revisiting PointNet++ with Improved Training and Scaling Strategiescited by 436
  33. 2017PixelCNN++: Improving the PixelCNN with Discretized Logistic Mixture Likelihood and Other Modificationscited by 417
  34. 2022Rethinking Network Design and Local Geometry in Point Cloud: A Simple Residual MLP Frameworkcited by 379
  35. 2021ABO: Dataset and Benchmarks for Real-World 3D Object Understandingcited by 366
  36. 2023OmniObject3D: Large-Vocabulary 3D Object Dataset for Realistic Perception, Reconstruction and Generationcited by 326
  37. 2019On Mutual Information Maximization for Representation Learningcited by 316
  38. 20203D-FUTURE: 3D Furniture shape with TextUREcited by 311

50 most cited of 80 All 80 (JSON)

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2023Self-supervised adversarial masking for 3D point cloud representation learning
  2. 2024GeoMask3D: Geometrically Informed Mask Selection for Self-Supervised Point Cloud Learning in 3D
  3. 2022Distillation with Contrast is All You Need for Self-Supervised Point Cloud Representation Learning
  4. 2022Self-Supervised Learning with Multi-View Rendering for 3D Point Cloud Analysis
  5. 2023Asymmetric Masked Distillation for Pre-Training Small Foundation Models
  6. 2025Foundry: Distilling 3D Foundation Models for the Edge
  7. 2025Sonata: Self-Supervised Learning of Reliable Point Representations
  8. 2026Emergent Multi-View Geometry Through Self-Distillation