mlsanityData to 7 Oct 2026

SynCo: Synthetic Hard Negatives for Contrastive Visual Representation Learning

arXiv 2410.02401 ↗First posted 3 Oct 2024cs.CV cs.AIcited by 5

Not on the map. Why?

Abstract (arXiv)

Contrastive learning relies on informative negatives to shape the representation space, yet obtaining hard negatives is costly, often requiring large batch sizes or extensive memory banks. We propose SynCo (Synthetic negatives in Contrastive learning), an approach that synthesizes hard negatives directly in the representation space from cached queue embeddings, with no additional forward passes or input-space processing. We find that six lightweight synthesis strategies, exhaustively covering the geometric, stochastic, and adversarial perturbation families, consistently improve learned representations at negligible computational cost. Although applicable to any InfoNCE-based contrastive objective, we demonstrate SynCo within the MoCo framework. On ImageNet ILSVRC-2012 linear evaluation at 200 epochs, SynCo yields improvements of +0.4% over MoCo-v2 and +1.0% over MoCHi. Unlike MoCHi, which degrades at extended pretraining schedules (underperforming MoCo-v2 by 2.4% at 800 epochs), SynCo does not: with a simple synthetic negative schedule, performance improves by +0.5% over MoCo-v2 at 800 epochs. SynCo also transfers well to a range of downstream tasks.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Builds on 49 of its 69 references that are in the map. First posted on arXiv 3 Oct 2024.

Builds on

49 of 69 references.

References · 49

  1. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  2. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
  3. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  4. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  5. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  6. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,155
  7. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
  8. 2019CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Featurescited by 2,875
  9. 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,845
  10. 2020Improved Baselines with Momentum Contrastive Learningcited by 2,282
  11. 2019RandAugment: Practical automated data augmentation with a reduced search spacecited by 2,261
  12. 2021Barlow Twins: Self-Supervised Learning via Redundancy Reductioncited by 1,818
37 more
  1. 2021An Empirical Study of Training Self-Supervised Vision Transformerscited by 1,610
  2. 2018Unsupervised Representation Learning by Predicting Image Rotationscited by 1,586
  3. 2020Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hyperspherecited by 1,393
  4. 2016Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzlescited by 1,379
  5. 2018Learning deep representations by mutual information estimation and maximizationcited by 1,265
  6. 2020What Makes for Good Views for Contrastive Learning?cited by 843
  7. 2019Self-Supervised Learning of Pretext-Invariant Representationscited by 806
  8. 2019Learning Representations by Maximizing Mutual Information Across Viewscited by 733
  9. 2020Prototypical Contrastive Learning of Unsupervised Representationscited by 636
  10. 2019A Theoretical Analysis of Contrastive Unsupervised Representation Learningcited by 530
  11. 2020Hard Negative Mixing for Contrastive Learningcited by 399
  12. 2021With a Little Help from My Friends: Nearest-Neighbor Contrastive Learning of Visual Representationscited by 324
  13. 2017Working hard to know your neighbor's margins: Local descriptor learning losscited by 212
  14. 2019Local Aggregation for Unsupervised Learning of Visual Embeddingscited by 204
  15. 2020Representation Learning via Invariant Causal Mechanismscited by 197
  16. 2021Aligning Pretraining for Detection via Object-Level Contrastive Learningcited by 110
  17. 2023Foundational Models Defining a New Era in Vision: A Survey and Outlookcited by 109
  18. 2021Generative Models as a Data Source for Multiview Representation Learningcited by 104
  19. 2021Contrastive Learning with Stronger Augmentationscited by 100
  20. 2021Instance Localization for Self-supervised Detection Pretrainingcited by 99
  21. 2020AdCo: Adversarial Contrast for Efficient Learning of Unsupervised Representations from Self-Trained Negative Adversariescited by 90
  22. 2021Divide and Contrast: Self-supervised Learning from Uncurated Datacited by 85
  23. 2021ReSSL: Relational Self-Supervised Learning with Weak Augmentationcited by 83
  24. 2022Pushing the limits of self-supervised ResNets: Can we outperform supervised learning without labels on ImageNet?cited by 67
  25. 2022Dual Temperature Helps Contrastive Learning Without Many Negative Samples: Towards Understanding and Simplifying MoCocited by 52
  26. 2021Exploring the Equivalence of Siamese Self-Supervised Learning via A Unified Gradient Frameworkcited by 49
  27. 2021Robust Contrastive Learning Using Negative Samples with Diminished Semanticscited by 43
  28. 2018Mining on Manifolds: Metric Learning without Labelscited by 36
  29. 2020SelfAugment: Automatic Augmentation Policies for Self-Supervised Learningcited by 32
  30. 2023Understanding the Role of the Projector in Knowledge Distillationcited by 31
  31. 2020EqCo: Equivalent Rules for Self-supervised Contrastive Learningcited by 12
  32. 2021Fundamental Tradeoffs in Distributionally Adversarial Trainingcited by 12
  33. 2022TiCo: Transformation Invariance and Covariance Contrast for Self-Supervised Visual Representation Learningcited by 12
  34. 2023All4One: Symbiotic Neighbour Contrastive Learning via Self-Attention and Redundancy Reductioncited by 11
  35. 2022CaCo: Both Positive and Negative Samples are Directly Learnable via Cooperative-adversarial Contrastive Learningcited by 9
  36. 2023Guarding Barlow Twins Against Overfitting with Mixed Samplescited by 7
  37. 2023Hard-Negative Sampling for Contrastive Learning: Optimal Representation Geometry and Neural- vs Dimensional-Collapsecited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 1; 2026: 4.

5 papers.

  1. 2026ViTAMINS: An Empirical Study of Training Self-Supervised Vision Transformers with Synthetic Hard Negativescited by 1
  2. 2025Unsupervised Training of Vision Transformers with Synthetic Negativescited by 0
  3. 2026DiffusionPrint: Learning Generative Fingerprints for Diffusion-Based Inpainting Localizationcited by 0
  4. 2026Three Necessary Principles for Self-Supervised Visual Representation Learningcited by 0
  5. 2026What Makes Synthetic Hard Negatives Work in Vision-Language Pretraining?cited by 0