# Use All The Labels: A Hierarchical Multi-Label Contrastive Learning Framework

arXiv: https://arxiv.org/abs/2204.13207 · HTML: https://mlsanity.com/paper/2204.13207/

## Key facts

- Cited by 59 papers in the map (data through 14 Oct 2026).
- Builds on 32 of its 52 references that are in the map.
- First posted on arXiv 27 Apr 2022.

## Builds on (most cited first, 30 of 32)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [FaceNet: A Unified Embedding for Face Recognition and Clustering](https://mlsanity.com/paper/1503.03832/) (2015)
- [Supervised Contrastive Learning](https://mlsanity.com/paper/2004.11362/) (2020)
- [Unsupervised Feature Learning via Non-Parametric Instance-level Discrimination](https://mlsanity.com/paper/1805.01978/) (2018)
- [Contrastive Multiview Coding](https://mlsanity.com/paper/1906.05849/) (2019)
- [Learning deep representations by mutual information estimation and maximization](https://mlsanity.com/paper/1808.06670/) (2018)
- [Unsupervised Visual Representation Learning by Context Prediction](https://mlsanity.com/paper/1505.05192/) (2015)
- [An overview of gradient descent optimization algorithms](https://mlsanity.com/paper/1609.04747/) (2016)
- [Data-Efficient Image Recognition with Contrastive Predictive Coding](https://mlsanity.com/paper/1905.09272/) (2019)
- [Prototypical Contrastive Learning of Unsupervised Representations](https://mlsanity.com/paper/2005.04966/) (2020)
- [Dense Contrastive Learning for Self-Supervised Visual Pre-Training](https://mlsanity.com/paper/2011.09157/) (2020)
- [A Metric Learning Reality Check](https://mlsanity.com/paper/2003.08505/) (2020)
- [On Mutual Information Maximization for Representation Learning](https://mlsanity.com/paper/1907.13625/) (2019)
- [Sampling Matters in Deep Embedding Learning](https://mlsanity.com/paper/1706.07567/) (2017)
- [Split-Brain Autoencoders: Unsupervised Learning by Cross-Channel Prediction](https://mlsanity.com/paper/1611.09842/) (2016)
- [VideoMoCo: Contrastive Video Representation Learning with Temporally Adversarial Examples](https://mlsanity.com/paper/2103.05905/) (2021)
- [Making Better Mistakes: Leveraging Class Hierarchies with Deep Networks](https://mlsanity.com/paper/1912.09393/) (2019)
- [Deep Metric Learning with Hierarchical Triplet Loss](https://mlsanity.com/paper/1810.06951/) (2018)
- [Image Representations Learned With Unsupervised Pre-Training Contain Human-like Biases](https://mlsanity.com/paper/2010.15052/) (2020)
- [MultiGrain: a unified image embedding for classes and instances](https://mlsanity.com/paper/1902.05509/) (2019)
- [Ranked List Loss for Deep Metric Learning](https://mlsanity.com/paper/1903.03238/) (2019)
- [Coherent Hierarchical Multi-Label Classification Networks](https://mlsanity.com/paper/2010.10151/) (2020)
- [Do Convolutional Neural Networks Learn Class Hierarchy?](https://mlsanity.com/paper/1710.06501/) (2017)
- [Multi-Label Contrastive Learning for Abstract Visual Reasoning](https://mlsanity.com/paper/2012.01944/) (2020)
- [Leveraging Class Hierarchies with Metric-Guided Prototype Learning](https://mlsanity.com/paper/2007.03047/) (2020)

## Built on by (most cited first, 30 of 59)

- [BioCLIP: A Vision Foundation Model for the Tree of Life](https://mlsanity.com/paper/2311.18803/) (2023)
- [OmniSeg3D: Omniversal 3D Segmentation via Hierarchical Contrastive Learning](https://mlsanity.com/paper/2311.11666/) (2023)
- [SelEx: Self-Expertise in Fine-Grained Generalized Category Discovery](https://mlsanity.com/paper/2408.14371/) (2024)
- [OpenCon: Open-world Contrastive Learning](https://mlsanity.com/paper/2208.02764/) (2022)
- [Multi-Label Guided Soft Contrastive Learning for Efficient Earth Observation Pretraining](https://mlsanity.com/paper/2405.20462/) (2024)
- [MaskCon: Masked Contrastive Learning for Coarse-Labelled Dataset](https://mlsanity.com/paper/2303.12756/) (2023)
- [TransHP: Image Classification with Hierarchical Prompting](https://mlsanity.com/paper/2304.06385/) (2023)
- [Visually Consistent Hierarchical Image Classification](https://mlsanity.com/paper/2406.11608/) (2024)
- [Hierarchical discriminative learning improves visual representations of biomedical microscopy](https://mlsanity.com/paper/2303.01605/) (2023)
- [An Effective Deployment of Contrastive Learning in Multi-label Text Classification](https://mlsanity.com/paper/2212.00552/) (2022)
- [QUEEN: Query Unlearning against Model Extraction](https://mlsanity.com/paper/2407.01251/) (2024)
- [Exploiting Contrastive Learning and Numerical Evidence for Confusing Legal Judgment Prediction](https://mlsanity.com/paper/2211.08238/) (2022)
- [End-to-End Supervised Multilabel Contrastive Learning](https://mlsanity.com/paper/2307.03967/) (2023)
- [Joint Multiple Intent Detection and Slot Filling with Supervised Contrastive Learning and Self-Distillation](https://mlsanity.com/paper/2308.14654/) (2023)
- [Introspective Deep Metric Learning](https://mlsanity.com/paper/2309.09982/) (2023)
- [HEX: Hierarchical Emergence Exploitation in Self-Supervised Algorithms](https://mlsanity.com/paper/2410.23200/) (2024)
- [SEAL: Semantic-Aware Hierarchical Learning for Generalized Category Discovery](https://mlsanity.com/paper/2510.18740/) (2025)
- [Revisiting Logistic-softmax Likelihood in Bayesian Meta-Learning for Few-Shot Classification](https://mlsanity.com/paper/2310.10379/) (2023)
- [Instances and Labels: Hierarchy-aware Joint Supervised Contrastive Learning for Hierarchical Multi-Label Text Classification](https://mlsanity.com/paper/2310.05128/) (2023)
- [Exploring Contrastive Learning for Long-Tailed Multi-Label Text Classification](https://mlsanity.com/paper/2404.08720/) (2024)
- [Class Prototypes based Contrastive Learning for Classifying Multi-Label and Fine-Grained Educational Videos](https://mlsanity.com/paper/2510.11204/) (2025)
- [Incorporating Domain Knowledge Graph into Multimodal Movie Genre Classification with Self-Supervised Attention and Contrastive Learning](https://mlsanity.com/paper/2310.08032/) (2023)
- [ProbMCL: Simple Probabilistic Contrastive Learning for Multi-label Visual Classification](https://mlsanity.com/paper/2401.01448/) (2024)
- [Learning Label Hierarchy with Supervised Contrastive Learning](https://mlsanity.com/paper/2402.00232/) (2024)
- [Language Model as Visual Explainer](https://mlsanity.com/paper/2412.07802/) (2024)
- [Can we make NeRF-based visual localization privacy-preserving?](https://mlsanity.com/paper/2508.18971/) (2025)
- [Robult: Leveraging Redundancy and Modality Specific Features for Robust Multimodal Learning](https://mlsanity.com/paper/2509.03477/) (2025)
- [Free-Grained Hierarchical Visual Recognition](https://mlsanity.com/paper/2510.14737/) (2025)
- [Multi-Similarity Contrastive Learning](https://mlsanity.com/paper/2307.02712/) (2023)
- [Acoustic identification of individual animals with hierarchical contrastive learning](https://mlsanity.com/paper/2409.08673/) (2024)

## Abstract (arXiv)

Current contrastive learning frameworks focus on leveraging a single supervisory signal to learn representations, which limits the efficacy on unseen data and downstream tasks. In this paper, we present a hierarchical multi-label representation learning framework that can leverage all available labels and preserve the hierarchical relationship between classes. We introduce novel hierarchy preserving losses, which jointly apply a hierarchical penalty to the contrastive loss, and enforce the hierarchy constraint. The loss function is data driven and automatically adapts to arbitrary multi-label structures. Experiments on several datasets show that our relationship-preserving embedding performs well on a variety of tasks and outperform the baseline supervised and self-supervised approaches. Code is available at https://github.com/salesforce/hierarchicalContrastiveLearning.