# Energy-Based Contrastive Learning of Visual Representations

arXiv: https://arxiv.org/abs/2202.04933 · HTML: https://mlsanity.com/paper/2202.04933/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 45 references that are in the map.
- First posted on arXiv 10 Feb 2022.

## Builds on (most cited first, 24 of 24)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [Bootstrap your own latent: A new approach to self-supervised Learning](https://mlsanity.com/paper/2006.07733/) (2020)
- [Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms](https://mlsanity.com/paper/1708.07747/) (2017)
- [Exploring Simple Siamese Representation Learning](https://mlsanity.com/paper/2011.10566/) (2020)
- [Improved Baselines with Momentum Contrastive Learning](https://mlsanity.com/paper/2003.04297/) (2020)
- [Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hypersphere](https://mlsanity.com/paper/2005.10242/) (2020)
- [Contrastive Multiview Coding](https://mlsanity.com/paper/1906.05849/) (2019)
- [What Makes for Good Views for Contrastive Learning?](https://mlsanity.com/paper/2005.10243/) (2020)
- [Your Classifier is Secretly an Energy Based Model and You Should Treat it Like One](https://mlsanity.com/paper/1912.03263/) (2019)
- [How to Train Your Energy-Based Models](https://mlsanity.com/paper/2101.03288/) (2021)
- [Contrastive Learning Inverts the Data Generating Process](https://mlsanity.com/paper/2102.08850/) (2021)
- [Telescoping Density-Ratio Estimation](https://mlsanity.com/paper/2006.12204/) (2020)
- [Flow Contrastive Estimation of Energy-Based Models](https://mlsanity.com/paper/1912.00589/) (2019)
- [No MCMC for me: Amortized sampling for fast and stable training of energy-based models](https://mlsanity.com/paper/2010.04230/) (2020)
- [Joint Generative and Contrastive Learning for Unsupervised Person Re-identification](https://mlsanity.com/paper/2012.09071/) (2020)
- [Unsupervised Feature Learning by Cross-Level Instance-Group Discrimination](https://mlsanity.com/paper/2008.03813/) (2020)
- [Understanding Hard Negatives in Noise Contrastive Estimation](https://mlsanity.com/paper/2104.06245/) (2021)
- [JEM++: Improved Techniques for Training JEM](https://mlsanity.com/paper/2109.09032/) (2021)
- [How to Train Your Energy-Based Model for Regression](https://mlsanity.com/paper/2005.01698/) (2020)

## Built on by (most cited first, 13 of 13)

- [Artificial General Intelligence (AGI)-Native Wireless Systems: A Journey Beyond 6G](https://mlsanity.com/paper/2405.02336/) (2024)
- [Learning Symbolic Representations Through Joint GEnerative and DIscriminative Training](https://mlsanity.com/paper/2304.11357/) (2023)
- [Collapse-Proof Non-Contrastive Self-Supervised Learning](https://mlsanity.com/paper/2410.04959/) (2024)
- [GEDI: GEnerative and DIscriminative Training for Self-Supervised Learning](https://mlsanity.com/paper/2212.13425/) (2022)
- [EQ-CBM: A Probabilistic Concept Bottleneck with Energy-based Models and Quantized Vectors](https://mlsanity.com/paper/2409.14630/) (2024)
- [The Triad of Failure Modes and a Possible Way Out](https://mlsanity.com/paper/2309.15420/) (2023)
- [Graph Structure Refinement with Energy-based Contrastive Learning](https://mlsanity.com/paper/2412.17856/) (2024)
- [Unifying Self-Supervised Clustering and Energy-Based Models](https://mlsanity.com/paper/2401.00873/) (2023)
- [Do You Know What You Are Talking About? Characterizing Query-Knowledge Relevance For Reliable Retrieval Augmented Generation](https://mlsanity.com/paper/2410.08320/) (2024)
- [On Discriminative Probabilistic Modeling for Self-Supervised Representation Learning](https://mlsanity.com/paper/2410.09156/) (2024)
- [Energy-based Domain-Adaptive Segmentation with Depth Guidance](https://mlsanity.com/paper/2402.03795/) (2024)
- [Feature Augmentation for Self-supervised Contrastive Learning: A Closer Look](https://mlsanity.com/paper/2410.12396/) (2024)
- [A Tutorial on Discriminative Clustering and Mutual Information](https://mlsanity.com/paper/2505.04484/) (2025)

## Abstract (arXiv)

Contrastive learning is a method of learning visual representations by training Deep Neural Networks (DNNs) to increase the similarity between representations of positive pairs (transformations of the same image) and reduce the similarity between representations of negative pairs (transformations of different images). Here we explore Energy-Based Contrastive Learning (EBCLR) that leverages the power of generative learning by combining contrastive learning with Energy-Based Models (EBMs). EBCLR can be theoretically interpreted as learning the joint distribution of positive pairs, and it shows promising results on small and medium-scale datasets such as MNIST, Fashion-MNIST, CIFAR-10, and CIFAR-100. Specifically, we find EBCLR demonstrates from X4 up to X20 acceleration compared to SimCLR and MoCo v2 in terms of training epochs. Furthermore, in contrast to SimCLR, we observe EBCLR achieves nearly the same performance with 254 negative pairs (batch size 128) and 30 negative pairs (batch size 16) per positive pair, demonstrating the robustness of EBCLR to small numbers of negative pairs. Hence, EBCLR provides a novel avenue for improving contrastive learning methods that usually require large datasets with a significant number of negative pairs per iteration to achieve reasonable performance on downstream tasks. Code: https://github.com/1202kbs/EBCLR