mlsanityData to 14 Oct 2026

MOCA: Self-supervised Representation Learning by Predicting Masked Online Codebook Assignments

arXiv 2307.09361 ↗First posted 18 Jul 2023cs.CV cs.AI cs.LGcited by 9

Not on the map. Why?

Abstract (arXiv)

Self-supervised learning can be used for mitigating the greedy needs of Vision Transformer networks for very large fully-annotated datasets. Different classes of self-supervised learning offer representations with either good contextual reasoning properties, e.g., using masked image modeling strategies, or invariance to image perturbations, e.g., with contrastive methods. In this work, we propose a single-stage and standalone method, MOCA, which unifies both desired properties using novel mask-and-predict objectives defined with high-level features (instead of pixel-level details). Moreover, we show how to effectively employ both learning paradigms in a synergistic and computation-efficient way. Doing so, we achieve new state-of-the-art results on low-shot settings and strong experimental results in various evaluation protocols with a training that is at least 3 times faster than prior methods. We provide the implementation code at https://github.com/valeoai/MOCA.

Key facts. Cited by 9 papers in the map (data through 14 Oct 2026). Builds on 77 of its 89 references that are in the map. First posted on arXiv 18 Jul 2023.

Builds on

77 of 89 references.

References · 77

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  5. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  6. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  7. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,498
  8. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  9. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,513
  10. 2020End-to-End Object Detection with Transformerscited by 7,414
  11. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
  12. 2017Mask R-CNNcited by 7,277
38 more
  1. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,968
  2. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,475
  3. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,183
  4. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 5,045
  5. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,750
  6. 2020Training data-efficient image transformers & distillation through attentioncited by 4,586
  7. 2016The Cityscapes Dataset for Semantic Urban Scene Understandingcited by 4,444
  8. 2021Zero-Shot Text-to-Image Generationcited by 4,288
  9. 2021On the Opportunities and Risks of Foundation Modelscited by 4,033
  10. 2019CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Featurescited by 2,884
  11. 2020Exploring Simple Siamese Representation Learningcited by 2,866
  12. 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,857
  13. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,375
  14. 2020Improved Baselines with Momentum Contrastive Learningcited by 2,287
  15. 2019RandAugment: Practical automated data augmentation with a reduced search spacecited by 2,269
  16. 2021Barlow Twins: Self-Supervised Learning via Redundancy Reductioncited by 1,838
  17. 2021An Empirical Study of Training Self-Supervised Vision Transformerscited by 1,616
  18. 2018Unsupervised Representation Learning by Predicting Image Rotationscited by 1,591
  19. 2016Context Encoders: Feature Learning by Inpaintingcited by 1,428
  20. 2016Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzlescited by 1,382
  21. 2021SimMIM: A Simple Framework for Masked Image Modelingcited by 1,153
  22. 2021Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNetcited by 1,101
  23. 2015Unsupervised Visual Representation Learning by Context Predictioncited by 1,066
  24. 2021VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learningcited by 1,057
  25. 2018Deep Clustering for Unsupervised Learning of Visual Featurescited by 1,028
  26. 2021Scaling Vision Transformerscited by 999
  27. 2016Deep Networks with Stochastic Depthcited by 979
  28. 2020On Layer Normalization in the Transformer Architecturecited by 922
  29. 2021iBOT: Image BERT Pre-Training with Online Tokenizercited by 903
  30. 2021Segmenter: Transformer for Semantic Segmentationcited by 855
  31. 2020What Makes for Good Views for Contrastive Learning?cited by 846
  32. 2019Self-Supervised Learning of Pretext-Invariant Representationscited by 808
  33. 2021Going deeper with Image Transformerscited by 764
  34. 2019Learning Representations by Maximizing Mutual Information Across Viewscited by 734
  35. 2022data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Languagecited by 727
  36. 2021Masked Feature Prediction for Self-Supervised Visual Pre-Trainingcited by 576
  37. 2019Self-labelling via simultaneous clustering and representation learningcited by 528
  38. 2019Cascade R-CNN: High Quality Object Detection and Instance Segmentationcited by 429

50 most cited of 77 All 77 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 2; 2025: 6; 2026: 1.

9 papers.

  1. 2024DINO-Foresight: Looking into the Future with DINOcited by 45
  2. 2025Boosting Generative Image Modeling via Joint Image-Feature Synthesiscited by 45
  3. 2025Franca: Nested Matryoshka Clustering for Scalable Visual Representation Learningcited by 21
  4. 2024OccFeat: Self-supervised Occupancy Feature Prediction for Pretraining BEV Segmentation Networkscited by 17
  5. 2025Cluster and Predict Latent Patches for Improved Masked Image Modelingcited by 17
  6. 2025DIP: Unsupervised Dense In-Context Post-training of Visual Representationscited by 3
  7. 2025Unleashing the Intrinsic Visual Representation Capability of Multimodal Large Language Modelscited by 2
  8. 2026Boosting Visual Instruction Tuning with Self-Supervised Guidancecited by 1
  9. 2025Why Prototypes Collapse: Diagnosing and Preventing Partial Collapse in Prototypical Self-Supervised Learningcited by 0