mlsanityData to 7 Oct 2026

Exponential Moving Average Normalization for Self-supervised and Semi-supervised Learning

arXiv 2101.08482 ↗First posted 21 Jan 2021cs.LG cs.AI cs.CVcited by 78

Not on the map. Why?

Abstract (arXiv)

We present a plug-in replacement for batch normalization (BN) called exponential moving average normalization (EMAN), which improves the performance of existing student-teacher based self- and semi-supervised learning techniques. Unlike the standard BN, where the statistics are computed within each batch, EMAN, used in the teacher, updates its statistics by exponential moving average from the BN statistics of the student. This design reduces the intrinsic cross-sample dependency of BN and enhances the generalization of the teacher. EMAN improves strong baselines for self-supervised learning by 4-6/1-2 points and semi-supervised learning by about 7/2 points, when 1%/10% supervised labels are available on ImageNet. These improvements are consistent across methods, network architectures, training duration, and datasets, demonstrating the general effectiveness of this technique. The code is available at https://github.com/amazon-research/exponential-moving-average-normalization.

Key facts. Cited by 78 papers in the map (data through 7 Oct 2026). Builds on 40 of its 53 references that are in the map. First posted on arXiv 21 Jan 2021.

Builds on

40 of 53 references.

References · 40

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  5. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  6. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  7. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  8. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  9. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  10. 2016Layer Normalizationcited by 5,579
  11. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
  12. 2015ShapeNet: An Information-Rich 3D Model Repositorycited by 2,880
28 more
  1. 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,845
  2. 2020Improved Baselines with Momentum Contrastive Learningcited by 2,282
  3. 2020FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidencecited by 2,080
  4. 2018Unsupervised Feature Learning via Non-Parametric Instance-level Discriminationcited by 1,838
  5. 2018Unsupervised Representation Learning by Predicting Image Rotationscited by 1,586
  6. 2018Group Normalizationcited by 1,514
  7. 2016Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzlescited by 1,379
  8. 2020Big Self-Supervised Models are Strong Semi-Supervised Learnerscited by 1,375
  9. 2019Self-training with Noisy Student improves ImageNet classificationcited by 1,304
  10. 2019Contrastive Multiview Codingcited by 1,297
  11. 2018Averaging Weights Leads to Wider Optima and Better Generalizationcited by 1,194
  12. 2016Instance Normalization: The Missing Ingredient for Fast Stylizationcited by 1,187
  13. 2016Colorful Image Colorizationcited by 1,113
  14. 2019Unsupervised Data Augmentation for Consistency Trainingcited by 1,057
  15. 2016Temporal Ensembling for Semi-Supervised Learningcited by 1,054
  16. 2017Virtual Adversarial Training: A Regularization Method for Supervised and Semi-Supervised Learningcited by 918
  17. 2019Self-Supervised Learning of Pretext-Invariant Representationscited by 806
  18. 2019Data-Efficient Image Recognition with Contrastive Predictive Codingcited by 672
  19. 2019Self-supervised Visual Feature Learning with Deep Neural Networks: A Surveycited by 656
  20. 2020Prototypical Contrastive Learning of Unsupervised Representationscited by 636
  21. 2017Snapshot Ensembles: Train 1, get M for freecited by 349
  22. 2019S4L: Self-Supervised Semi-Supervised Learningcited by 335
  23. 2019Local Aggregation for Unsupervised Learning of Visual Embeddingscited by 204
  24. 2019Scaling and Benchmarking Self-Supervised Visual Representation Learningcited by 201
  25. 2018There Are Many Consistent Explanations of Unlabeled Data: Why You Should Averagecited by 82
  26. 2019Central Similarity Quantization for Efficient Image and Video Retrievalcited by 77
  27. 2018MEAL: Multi-Model Ensemble via Adversarial Learningcited by 52
  28. 2020Towards Stabilizing Batch Statistics in Backward Propagation of Batch Normalizationcited by 19

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 12; 2022: 18; 2023: 24; 2024: 12; 2025: 8; 2026: 4.

78 papers.

Sort:

By area:

  1. 2023ZipIt! Merging Models from Different Tasks without Trainingcited by 178
  2. 2022SimMatch: Semi-supervised Learning with Similarity Matchingcited by 118
  3. 2021Perturbed and Strict Mean Teachers for Semi-supervised Semantic Segmentationcited by 113
  4. 2022Debiased Learning from Naturally Imbalanced Pseudo-Labelscited by 85
  5. 2024Exponential Moving Average of Weights in Deep Learning: Dynamics and Benefitscited by 65
  6. 2021Unsupervised Domain Adaptive 3D Detection with Multi-Level Consistencycited by 63
  7. 2022Semi-supervised Vision Transformers at Scalecited by 49
  8. 2021Rethinking Pseudo Labels for Semi-Supervised Object Detectioncited by 45
  9. 2021Rethinking "Batch" in BatchNormcited by 44
  10. 2021Multi-Layer Pseudo-Supervision for Histopathology Tissue Semantic Segmentation using Patch-level Classification Labelscited by 43
  11. 2022On the Importance of Asymmetry for Siamese Representation Learningcited by 38
  12. 2022Autoregressive 3D Shape Generation via Canonical Mappingcited by 37
38 more
  1. 2023Diffusion Models and Semi-Supervised Learners Benefit Mutually with Few Labelscited by 37
  2. 2021Data-Free Knowledge Transfer: A Surveycited by 31
  3. 2022Learning Where to Learn in Cross-View Self-Supervised Learningcited by 31
  4. 2022Omni-DETR: Omni-Supervised Object Detection with Transformerscited by 28
  5. 2021Unsupervised Selective Labeling for More Effective Semi-Supervised Learningcited by 27
  6. 2023CiPR: An Efficient Framework with Cross-instance Positive Relations for Generalized Category Discoverycited by 26
  7. 2021Multimodal Semi-Supervised Learning for 3D Objectscited by 24
  8. 2022CMW-Net: Learning a Class-Aware Sample Weighting Mapping for Robust Deep Learningcited by 24
  9. 2021Hybrid Dynamic Contrast and Probability Distillation for Unsupervised Person Re-Idcited by 22
  10. 2023Semi-supervised learning made simple with self-supervised clusteringcited by 20
  11. 2021Barely-Supervised Learning: Semi-Supervised Learning with very few labeled imagescited by 17
  12. 2022A soft nearest-neighbor framework for continual semi-supervised learningcited by 17
  13. 2023Shrinking Class Space for Enhanced Certainty in Semi-Supervised Learningcited by 17
  14. 2023Merging Vision Transformers from Different Tasks and Domainscited by 17
  15. 2024Training Neural Networks from Scratch with Parallel Low-Rank Adapterscited by 17
  16. 2024ProtoOcc: Accurate, Efficient 3D Occupancy Prediction Using Dual Branch Encoder-Prototype Query Decodercited by 17
  17. 2023Bytes Are All You Need: Transformers Operating Directly On File Bytescited by 16
  18. 2023Exploiting Unlabelled Photos for Stronger Fine-Grained SBIRcited by 14
  19. 2023ProtoCon: Pseudo-label Refinement via Online Clustering and Prototypical Consistency for Efficient Semi-supervised Learningcited by 13
  20. 2023SimMatchV2: Semi-Supervised Learning with Graph Consistencycited by 12
  21. 2021RegionCL: Can Simple Region Swapping Contribute to Contrastive Learning?cited by 11
  22. 2023A New Benchmark: On the Utility of Synthetic Data with Blender for Bare Supervised Learning and Downstream Domain Adaptationcited by 11
  23. 2023Boosting Semi-Supervised Learning by bridging high and low-confidence predictionscited by 11
  24. 2022On the Pros and Cons of Momentum Encoder in Self-Supervised Visual Representation Learningcited by 10
  25. 2022MaxMatch: Semi-Supervised Learning with Worst-Case Consistencycited by 9
  26. 2024Flipped Classroom: Aligning Teacher Attention with Student in Generalized Category Discoverycited by 9
  27. 2022Bootstrapped Representation Learning for Skeleton-Based Action Recognitioncited by 8
  28. 2023SequenceMatch: Revisiting the design of weak-strong augmentations for Semi-supervised learningcited by 8
  29. 2022Self-Supervised Visual Representation Learning via Residual Momentumcited by 7
  30. 2025Self-Reinforcing Prototype Evolution with Dual-Knowledge Cooperation for Semi-Supervised Lifelong Person Re-Identificationcited by 7
  31. 2023Robust Domain Adaptive Object Detection with Unified Multi-Granularity Alignmentcited by 6
  32. 2023SemPPL: Predicting pseudo-labels for better contrastive representationscited by 6
  33. 2025CoA: Towards Real Image Dehazing via Compression-and-Adaptationcited by 6
  34. 2023Transfer Learning for Fine-grained Classification Using Semi-supervised Learning and Visual Transformerscited by 5
  35. 2022Active Self-Semi-Supervised Learning for Few Labeled Samplescited by 4
  36. 2022Dense FixMatch: a simple semi-supervised learning method for pixel-wise prediction taskscited by 4
  37. 2022DA-CIL: Towards Domain Adaptive Class-Incremental 3D Object Detectioncited by 4
  38. 2024FedADMM-InSa: An Inexact and Self-Adaptive ADMM for Federated Learningcited by 4

50 most cited of 78 All 78 (JSON)