ViTAMINS: An Empirical Study of Training Self-Supervised Vision Transformers with Synthetic Hard Negatives
Not on the map. Why?
Abstract (arXiv)
We introduce ViTAMINS, a method that integrates synthetic hard negatives into unsupervised vision transformer pretraining to improve representation quality. Our approach is thoroughly benchmarked on ImageNet and transfer learning, image retrieval, copy detection, and image, video segmentation tasks. Notably, our proposed negatives give rise to emergent properties, where learned representations contain explicit information about the semantic content of an image and serve as excellent classifiers (up to +11.3% over baselines). ViTAMINS achieves these benefits through simple modifications to existing contrastive frameworks and outperforms competing methods while being more resource efficient, e.g., our ViT-B surpasses V-JEPA with ViT-L. Our findings motivate reconsidering contrastive learning as a simpler yet powerful alternative to dominant generative and self-distillation approaches.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 49 of its 68 references that are in the map. First posted on arXiv 1 Sep 2026.
Builds on
49 of 68 references.
References · 49
- 2017Attention Is All You Needcited by 59,932
- 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
- 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
- 2020Language Models are Few-Shot Learnerscited by 32,199
- 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
- 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
- 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,764
- 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
- 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
- 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
- 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
- 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
37 more
- 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
- 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
- 2020Exploring Simple Siamese Representation Learningcited by 2,853
- 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,845
- 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
- 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,360
- 2020Improved Baselines with Momentum Contrastive Learningcited by 2,282
- 2013Describing Textures in the Wildcited by 2,110
- 2021Barlow Twins: Self-Supervised Learning via Redundancy Reductioncited by 1,818
- 2021An Empirical Study of Training Self-Supervised Vision Transformerscited by 1,610
- 2018Unsupervised Representation Learning by Predicting Image Rotationscited by 1,586
- 2016Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzlescited by 1,379
- 2018Unified Perceptual Parsing for Scene Understandingcited by 1,265
- 2021SimMIM: A Simple Framework for Masked Image Modelingcited by 1,147
- 2016Semantic Understanding of Scenes through the ADE20K Datasetcited by 1,063
- 2018Deep Clustering for Unsupervised Learning of Visual Featurescited by 1,020
- 2023Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecturecited by 982
- 2020What Makes for Good Views for Contrastive Learning?cited by 843
- 2020Contrastive Learning with Hard Negative Samplescited by 619
- 2024Revisiting Feature Prediction for Learning Visual Representations from Videocited by 490
- 2019Cascade R-CNN: High Quality Object Detection and Instance Segmentationcited by 428
- 2020Hard Negative Mixing for Contrastive Learningcited by 399
- 2021With a Little Help from My Friends: Nearest-Neighbor Contrastive Learning of Visual Representationscited by 324
- 2022BEiT v2: Masked Image Modeling with Vector-Quantized Visual Tokenizerscited by 298
- 2022Context Autoencoder for Self-Supervised Representation Learningcited by 297
- 2020Whitening for Self-Supervised Representation Learningcited by 278
- 2020Space-Time Correspondence as a Contrastive Random Walkcited by 239
- 2025LeJEPA: Provable and Scalable Self-Supervised Learning Without the Heuristicscited by 227
- 2021Self-supervised Pretraining of Visual Features in the Wildcited by 197
- 2018Revisiting Oxford and Paris: Large-Scale Image Retrieval Benchmarkingcited by 189
- 2021Self-Supervised Learning with Swin Transformerscited by 121
- 2022Pushing the limits of self-supervised ResNets: Can we outperform supervised learning without labels on ImageNet?cited by 67
- 2023Learning Efficient Coding of Natural Images with Maximum Manifold Capacity Representationscited by 31
- 2022Unsupervised Visual Representation Learning by Synchronous Momentum Groupingcited by 22
- 2026Rectified LpJEPA: Joint-Embedding Predictive Architectures with Sparse and Maximum-Entropy Representationscited by 8
- 2024SynCo: Synthetic Hard Negatives for Contrastive Visual Representation Learningcited by 5
- 2024From Overfitting to Robustness: Quantity, Quality, and Variety Oriented Negative Sample Selection in Graph Contrastive Learningcited by 4
Built on by
Citing papers per year: 2026: 1.
1 paper.