# Not All Negatives are Equal: Label-Aware Contrastive Loss for Fine-grained Text Classification

arXiv: https://arxiv.org/abs/2109.05427 · HTML: https://mlsanity.com/paper/2109.05427/

## Key facts

- Cited by 43 papers in the map (data through 14 Oct 2026).
- Builds on 40 of its 56 references that are in the map.
- First posted on arXiv 12 Sep 2021.

## Builds on (most cited first, 30 of 40)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Supervised Contrastive Learning](https://mlsanity.com/paper/2004.11362/) (2020)
- [SimCSE: Simple Contrastive Learning of Sentence Embeddings](https://mlsanity.com/paper/2104.08821/) (2021)
- [Contrastive Multiview Coding](https://mlsanity.com/paper/1906.05849/) (2019)
- [Co-teaching: Robust Training of Deep Neural Networks with Extremely Noisy Labels](https://mlsanity.com/paper/1804.06872/) (2018)
- [Self-Supervised Learning of Pretext-Invariant Representations](https://mlsanity.com/paper/1912.01991/) (2019)
- [EDA: Easy Data Augmentation Techniques for Boosting Performance on Text Classification Tasks](https://mlsanity.com/paper/1901.11196/) (2019)
- [Sequence-Level Knowledge Distillation](https://mlsanity.com/paper/1606.07947/) (2016)
- [Learning Representations by Maximizing Mutual Information Across Views](https://mlsanity.com/paper/1906.00910/) (2019)
- [Learning to Reweight Examples for Robust Deep Learning](https://mlsanity.com/paper/1803.09050/) (2018)
- [Recipes for building an open-domain chatbot](https://mlsanity.com/paper/2004.13637/) (2020)
- [Learning from Noisy Labels with Deep Neural Networks: A Survey](https://mlsanity.com/paper/2007.08199/) (2020)
- [Patient Knowledge Distillation for BERT Model Compression](https://mlsanity.com/paper/1908.09355/) (2019)
- [GoEmotions: A Dataset of Fine-Grained Emotions](https://mlsanity.com/paper/2005.00547/) (2020)
- [Keystroke Dynamics as Part of Lifelogging](https://mlsanity.com/paper/2010.10144/) (2020)
- [Combating noisy labels by agreement: A joint training method with co-regularization](https://mlsanity.com/paper/2003.02752/) (2020)
- [Supervised Contrastive Learning for Pre-trained Language Model Fine-tuning](https://mlsanity.com/paper/2011.01403/) (2020)
- [CERT: Contrastive Self-supervised Learning for Language Understanding](https://mlsanity.com/paper/2005.12766/) (2020)
- [Evaluation of Output Embeddings for Fine-Grained Image Classification](https://mlsanity.com/paper/1409.8403/) (2014)
- [Active Bias: Training More Accurate Neural Networks by Emphasizing High Variance Samples](https://mlsanity.com/paper/1704.07433/) (2017)
- [Fine-Grained Analysis of Propaganda in News Articles](https://mlsanity.com/paper/1910.02517/) (2019)
- [COCO-LM: Correcting and Contrasting Text Sequences for Language Model Pretraining](https://mlsanity.com/paper/2102.08473/) (2021)
- [Fine-grained Fact Verification with Kernel Graph Attention Network](https://mlsanity.com/paper/1910.09796/) (2019)
- [Knowledge Distillation from Internal Representations](https://mlsanity.com/paper/1910.03723/) (2019)
- [Contrastive Distillation on Intermediate Representations for Language Model Compression](https://mlsanity.com/paper/2009.14167/) (2020)
- [SKEP: Sentiment Knowledge Enhanced Pre-training for Sentiment Analysis](https://mlsanity.com/paper/2005.05635/) (2020)

## Built on by (most cited first, 30 of 43)

- [AnglE-optimized Text Embeddings](https://mlsanity.com/paper/2309.12871/) (2023)
- [Large Language Models Produce Responses Perceived to be Empathic](https://mlsanity.com/paper/2403.18148/) (2024)
- [Facilitating Contrastive Learning of Discourse Relational Senses by Exploiting the Hierarchy of Sense Relations](https://mlsanity.com/paper/2301.02724/) (2023)
- [A Survey of Methods for Addressing Class Imbalance in Deep-Learning Based Natural Language Processing](https://mlsanity.com/paper/2210.04675/) (2022)
- [Recent advances in text embedding: A Comprehensive Review of Top-Performing Methods on the MTEB Benchmark](https://mlsanity.com/paper/2406.01607/) (2024)
- [CPL: Counterfactual Prompt Learning for Vision and Language Models](https://mlsanity.com/paper/2210.10362/) (2022)
- [Contrastive Representation Regularization for Vision-Language-Action Models](https://mlsanity.com/paper/2510.01711/) (2025)
- [Fine-grained Category Discovery under Coarse-grained supervision with Hierarchical Weighted Self-contrastive Learning](https://mlsanity.com/paper/2210.07733/) (2022)
- [Label-Aware Hyperbolic Embeddings for Fine-grained Emotion Classification](https://mlsanity.com/paper/2306.14822/) (2023)
- [A Multi-level Supervised Contrastive Learning Framework for Low-Resource Natural Language Inference](https://mlsanity.com/paper/2205.15550/) (2022)
- [Exploiting Contrastive Learning and Numerical Evidence for Confusing Legal Judgment Prediction](https://mlsanity.com/paper/2211.08238/) (2022)
- [Simple-Sampling and Hard-Mixup with Prototypes to Rebalance Contrastive Learning for Text Classification](https://mlsanity.com/paper/2405.11524/) (2024)
- [Label Anchored Contrastive Learning for Language Understanding](https://mlsanity.com/paper/2205.10227/) (2022)
- [Subjective Crowd Disagreements for Subjective Data: Uncovering Meaningful CrowdOpinion with Population-level Learning](https://mlsanity.com/paper/2307.10189/) (2023)
- [Joint Multiple Intent Detection and Slot Filling with Supervised Contrastive Learning and Self-Distillation](https://mlsanity.com/paper/2308.14654/) (2023)
- [CL-XABSA: Contrastive Learning for Cross-lingual Aspect-based Sentiment Analysis](https://mlsanity.com/paper/2204.00791/) (2022)
- [Using Positive Matching Contrastive Loss with Facial Action Units to mitigate bias in Facial Expression Recognition](https://mlsanity.com/paper/2303.04896/) (2023)
- [W-procer: Weighted Prototypical Contrastive Learning for Medical Few-Shot Named Entity Recognition](https://mlsanity.com/paper/2305.18624/) (2023)
- [Controlled Text Generation with Hidden Representation Transformations](https://mlsanity.com/paper/2305.19230/) (2023)
- [Instances and Labels: Hierarchy-aware Joint Supervised Contrastive Learning for Hierarchical Multi-Label Text Classification](https://mlsanity.com/paper/2310.05128/) (2023)
- [DNA: Denoised Neighborhood Aggregation for Fine-grained Category Discovery](https://mlsanity.com/paper/2310.10151/) (2023)
- [The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages](https://mlsanity.com/paper/2310.14557/) (2023)
- [Learning Label Hierarchy with Supervised Contrastive Learning](https://mlsanity.com/paper/2402.00232/) (2024)
- [Type-enriched Hierarchical Contrastive Strategy for Fine-Grained Entity Typing](https://mlsanity.com/paper/2208.10081/) (2022)
- [A Benchmark Study of Contrastive Learning for Arabic Social Meaning](https://mlsanity.com/paper/2210.12314/) (2022)
- [CoDiEmb: A Collaborative yet Distinct Framework for Unified Representation Learning in Information Retrieval and Semantic Textual Similarity](https://mlsanity.com/paper/2508.11442/) (2025)
- [Contrastive Learning of Sociopragmatic Meaning in Social Media](https://mlsanity.com/paper/2203.07648/) (2022)
- [Dictionary-Assisted Supervised Contrastive Learning](https://mlsanity.com/paper/2210.15172/) (2022)
- [Constrained Multi-Layer Contrastive Learning for Implicit Discourse Relationship Recognition](https://mlsanity.com/paper/2409.13716/) (2024)
- [Semantic Adapter for Universal Text Embeddings: Diagnosing and Mitigating Negation Blindness to Enhance Universality](https://mlsanity.com/paper/2504.00584/) (2025)

## Abstract (arXiv)

Fine-grained classification involves dealing with datasets with larger number of classes with subtle differences between them. Guiding the model to focus on differentiating dimensions between these commonly confusable classes is key to improving performance on fine-grained tasks. In this work, we analyse the contrastive fine-tuning of pre-trained language models on two fine-grained text classification tasks, emotion classification and sentiment analysis. We adaptively embed class relationships into a contrastive objective function to help differently weigh the positives and negatives, and in particular, weighting closely confusable negatives more than less similar negative examples. We find that Label-aware Contrastive Loss outperforms previous contrastive methods, in the presence of larger number and/or more confusable classes, and helps models to produce output distributions that are more differentiated.