mlsanityData to 14 Oct 2026

Momentum^2 Teacher: Momentum Teacher with Momentum Statistics for Self-Supervised Learning

arXiv 2101.07525 ↗First posted 19 Jan 2021cs.LG cs.CVcited by 14

Not on the map. Why?

Abstract (arXiv)

In this paper, we present a novel approach, Momentum2 Teacher, for student-teacher based self-supervised learning. The approach performs momentum update on both network weights and batch normalization (BN) statistics. The teacher's weight is a momentum update of the student, and the teacher's BN statistics is a momentum update of those in history. The Momentum2 Teacher is simple and efficient. It can achieve the state of the art results (74.5%) under ImageNet linear evaluation protocol using small-batch size(eg, 128), without requiring large-batch training on special hardware like TPU or inefficient across GPU operation (eg, shuffling BN, synced BN). Our implementation and pre-trained models will be given on GitHubfootnotehttps://github.com/zengarden/momentum2-teacher.

Key facts. Cited by 14 papers in the map (data through 14 Oct 2026). Builds on 53 of its 64 references that are in the map. First posted on arXiv 19 Jan 2021.

Builds on

53 of 64 references.

References · 53

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  3. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  4. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  5. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,513
  6. 2017Mask R-CNNcited by 7,277
  7. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 7,005
  8. 2016Layer Normalizationcited by 5,669
  9. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 5,045
  10. 2016Feature Pyramid Networks for Object Detectioncited by 5,036
  11. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,750
  12. 2020Exploring Simple Siamese Representation Learningcited by 2,866
38 more
  1. 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,857
  2. 2020Improved Baselines with Momentum Contrastive Learningcited by 2,287
  3. 2020FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidencecited by 2,085
  4. 2018Unsupervised Feature Learning via Non-Parametric Instance-level Discriminationcited by 1,845
  5. 2019FCOS: Fully Convolutional One-Stage Object Detectioncited by 1,577
  6. 2018Group Normalizationcited by 1,524
  7. 2017Accurate, Large Minibatch SGD: Training ImageNet in 1 Hourcited by 1,492
  8. 2016Context Encoders: Feature Learning by Inpaintingcited by 1,428
  9. 2016Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzlescited by 1,382
  10. 2019Contrastive Multiview Codingcited by 1,301
  11. 2017Cascade R-CNN: Delving into High Quality Object Detectioncited by 1,276
  12. 2018Learning deep representations by mutual information estimation and maximizationcited by 1,273
  13. 2019LVIS: A Dataset for Large Vocabulary Instance Segmentationcited by 1,217
  14. 2016Instance Normalization: The Missing Ingredient for Fast Stylizationcited by 1,198
  15. 2016Colorful Image Colorizationcited by 1,114
  16. 2015Unsupervised Visual Representation Learning by Context Predictioncited by 1,066
  17. 2019Unsupervised Data Augmentation for Consistency Trainingcited by 1,060
  18. 2017Virtual Adversarial Training: A Regularization Method for Supervised and Semi-Supervised Learningcited by 920
  19. 2020What Makes for Good Views for Contrastive Learning?cited by 846
  20. 2019Self-Supervised Learning of Pretext-Invariant Representationscited by 808
  21. 2019Learning Representations by Maximizing Mutual Information Across Viewscited by 734
  22. 2019Data-Efficient Image Recognition with Contrastive Predictive Codingcited by 673
  23. 2020Prototypical Contrastive Learning of Unsupervised Representationscited by 640
  24. 2016Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networkscited by 543
  25. 2017Large Batch Training of Convolutional Networkscited by 535
  26. 2017Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning resultscited by 472
  27. 2019S4L: Self-Supervised Semi-Supervised Learningcited by 335
  28. 2014Discriminative Unsupervised Feature Learning with Exemplar Convolutional Neural Networkscited by 300
  29. 2019Large Scale Adversarial Representation Learningcited by 239
  30. 2016Split-Brain Autoencoders: Unsupervised Learning by Cross-Channel Predictioncited by 217
  31. 2019Local Aggregation for Unsupervised Learning of Visual Embeddingscited by 204
  32. 2017Multi-task Self-Supervised Visual Learningcited by 200
  33. 2016Learning Features by Watching Objects Movecited by 138
  34. 2017Batch Renormalization: Towards Reducing Minibatch Dependence in Batch-Normalized Modelscited by 128
  35. 2020BYOL works even without batch statisticscited by 95
  36. 2017MegDet: A Large Mini-Batch Object Detectorcited by 66
  37. 2019Online Normalization for Training Neural Networkscited by 29
  38. 2016Normalization Propagation: A Parametric Technique for Removing Internal Covariate Shift in Deep Networkscited by 20

50 most cited of 53 All 53 (JSON)

Built on by

Cited by year

Citing papers per year: 2021: 5; 2022: 7; 2023: 1; 2024: 1.

14 papers.

Sort:

By area:

  1. 2021Rethinking "Batch" in BatchNormcited by 45
  2. 2022On the Importance of Asymmetry for Siamese Representation Learningcited by 39
  3. 2023Multi-Mode Online Knowledge Distillation for Self-Supervised Visual Representation Learningcited by 20
  4. 2021Bag of Tricks and A Strong baseline for Image Copy Detectioncited by 14
  5. 2022On the Pros and Cons of Momentum Encoder in Self-Supervised Visual Representation Learningcited by 10
  6. 2021Temporal Knowledge Consistency for Unsupervised Visual Representation Learningcited by 8
  7. 2022Bootstrapped Representation Learning for Skeleton-Based Action Recognitioncited by 8
  8. 2022Semantics-Consistent Feature Search for Self-Supervised Visual Representation Learningcited by 8
  9. 2021D2LV: A Data-Driven and Local-Verification Approach for Image Copy Detectioncited by 7
  10. 2022Self-Supervised Visual Representation Learning via Residual Momentumcited by 7
  11. 2021Voxel-wise Cross-Volume Representation Learning for 3D Neuron Reconstructioncited by 4
  12. 2022Adversarial Contrastive Self-Supervised Learningcited by 2
2 more
  1. 2024MCA: Moment Channel Attention Networkscited by 2
  2. 2022DSPNet: Towards Slimmable Pretrained Networks based on Discriminative Self-supervised Learningcited by 0