# Non-negative Contrastive Learning

arXiv: https://arxiv.org/abs/2403.12459 · HTML: https://mlsanity.com/paper/2403.12459/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- Builds on 32 of its 49 references that are in the map.
- First posted on arXiv 19 Mar 2024.

## Builds on (most cited first, 30 of 32)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization](https://mlsanity.com/paper/1610.02391/) (2016)
- [Gaussian Error Linear Units (GELUs)](https://mlsanity.com/paper/1606.08415/) (2016)
- [Benchmarking Neural Network Robustness to Common Corruptions and Perturbations](https://mlsanity.com/paper/1903.12261/) (2019)
- [Supervised Contrastive Learning](https://mlsanity.com/paper/2004.11362/) (2020)
- [Categorical Reparameterization with Gumbel-Softmax](https://mlsanity.com/paper/1611.01144/) (2016)
- [ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness](https://mlsanity.com/paper/1811.12231/) (2018)
- [Learning Robust Global Representations by Penalizing Local Predictive Power](https://mlsanity.com/paper/1905.13549/) (2019)
- [Challenging Common Assumptions in the Unsupervised Learning of Disentangled Representations](https://mlsanity.com/paper/1811.12359/) (2018)
- [A Theoretical Analysis of Contrastive Unsupervised Representation Learning](https://mlsanity.com/paper/1902.09229/) (2019)
- [Variational Autoencoders and Nonlinear ICA: A Unifying Framework](https://mlsanity.com/paper/1907.04809/) (2019)
- [Provable Guarantees for Self-Supervised Deep Learning with Spectral Contrastive Loss](https://mlsanity.com/paper/2106.04156/) (2021)
- [Matryoshka Representation Learning](https://mlsanity.com/paper/2205.13147/) (2022)
- [Chaos is a Ladder: A New Theoretical Understanding of Contrastive Learning via Augmentation Overlap](https://mlsanity.com/paper/2203.13457/) (2022)
- [How Mask Matters: Towards Theoretical Understandings of Masked Autoencoders](https://mlsanity.com/paper/2210.08344/) (2022)
- [Measuring Disentanglement: A Review of Metrics](https://mlsanity.com/paper/2012.09276/) (2020)
- [Identifiability of deep generative models without auxiliary information](https://mlsanity.com/paper/2206.10044/) (2022)
- [Theory and Evaluation Metrics for Learning Disentangled Representations](https://mlsanity.com/paper/1908.09961/) (2019)
- [ContraNorm: A Contrastive Learning Perspective on Oversmoothing and Beyond](https://mlsanity.com/paper/2303.06562/) (2023)
- [On Identifiability of Nonnegative Matrix Factorization](https://mlsanity.com/paper/1709.00614/) (2017)
- [Residual Relaxation for Multi-view Representation Learning](https://mlsanity.com/paper/2110.15348/) (2021)
- [On the Generalization of Multi-modal Contrastive Learning](https://mlsanity.com/paper/2306.04272/) (2023)
- [Contrastive Learning Is Spectral Clustering On Similarity Graph](https://mlsanity.com/paper/2303.15103/) (2023)
- [Rethinking the Effect of Data Augmentation in Adversarial Contrastive Learning](https://mlsanity.com/paper/2303.01289/) (2023)
- [Rethinking Weak Supervision in Helping Contrastive Learning](https://mlsanity.com/paper/2306.04160/) (2023)
- [Identifiable Contrastive Learning with Automatic Feature Importance Discovery](https://mlsanity.com/paper/2310.18904/) (2023)

## Built on by (most cited first, 19 of 19)

- [Beyond Matryoshka: Revisiting Sparse Coding for Adaptive Representation](https://mlsanity.com/paper/2503.01776/) (2025)
- [On the Limits of Sparse Autoencoders: A Theoretical Framework and Reweighted Remedy](https://mlsanity.com/paper/2506.15963/) (2025)
- [Beyond Interpretability: The Gains of Feature Monosemanticity on Model Robustness](https://mlsanity.com/paper/2410.21331/) (2024)
- [Projection Head is Secretly an Information Bottleneck](https://mlsanity.com/paper/2503.00507/) (2025)
- [Collapse-Proof Non-Contrastive Self-Supervised Learning](https://mlsanity.com/paper/2410.04959/) (2024)
- [CSRv2: Unlocking Ultra-Sparse Embeddings](https://mlsanity.com/paper/2602.05735/) (2026)
- [An Augmentation-Aware Theory for Self-Supervised Contrastive Learning](https://mlsanity.com/paper/2505.22196/) (2025)
- [DWM: Separating World Effects from Actions in Latent World Models](https://mlsanity.com/paper/2607.18715/) (2026)
- [Beyond Spectral Decomposition: Bayesian Contrastive Learning and its Non-negative Formulation via Factor Analysis](https://mlsanity.com/paper/2407.21740/) (2024)
- [A Fresh Look at Generalized Category Discovery through Non-negative Matrix Factorization](https://mlsanity.com/paper/2410.21807/) (2024)
- [Difficult Examples Hurt Unsupervised Contrastive Learning: A Theoretical Perspective](https://mlsanity.com/paper/2501.01317/) (2025)
- [Enhancing Uncertainty Estimation and Interpretability via Bayesian Non-negative Decision Layer](https://mlsanity.com/paper/2505.22199/) (2025)
- [Sparse CLIP: Co-Optimizing Interpretability and Performance in Contrastive Learning](https://mlsanity.com/paper/2601.20075/) (2026)
- [Disentangled Generative Graph Representation Learning](https://mlsanity.com/paper/2408.13471/) (2024)
- [An Exterior Method for Nonnegative Matrix Factorization](https://mlsanity.com/paper/2605.19325/) (2026)
- [No More K-means: Single-Stage Sparse Coding for Efficient Multi-Vector Retrieval](https://mlsanity.com/paper/2605.30120/) (2026)
- [Eigenfunction Extraction for Ordered Representation Learning](https://mlsanity.com/paper/2510.24672/) (2025)
- [CCAR: Intrinsic Robustness as an Emergent Geometric Property](https://mlsanity.com/paper/2604.16861/) (2026)
- [Bayesian Gated Non-Negative Contrastive Learning](https://mlsanity.com/paper/2605.28441/) (2026)

## Abstract (arXiv)

Deep representations have shown promising performance when transferred to downstream tasks in a black-box manner. Yet, their inherent lack of interpretability remains a significant challenge, as these features are often opaque to human understanding. In this paper, we propose Non-negative Contrastive Learning (NCL), a renaissance of Non-negative Matrix Factorization (NMF) aimed at deriving interpretable features. The power of NCL lies in its enforcement of non-negativity constraints on features, reminiscent of NMF's capability to extract features that align closely with sample clusters. NCL not only aligns mathematically well with an NMF objective but also preserves NMF's interpretability attributes, resulting in a more sparse and disentangled representation compared to standard contrastive learning (CL). Theoretically, we establish guarantees on the identifiability and downstream generalization of NCL. Empirically, we show that these advantages enable NCL to outperform CL significantly on feature disentanglement, feature selection, as well as downstream classification tasks. At last, we show that NCL can be easily extended to other learning scenarios and benefit supervised learning as well. Code is available at https://github.com/PKU-ML/non_neg.