# Dice Loss for Data-imbalanced NLP Tasks

arXiv: https://arxiv.org/abs/1911.02855 · HTML: https://mlsanity.com/paper/1911.02855/

## Key facts

- Cited by 182 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 7 Nov 2019.

## Built on by (most cited first, 30 of 182)

- [AnomalyCLIP: Object-agnostic Prompt Learning for Zero-shot Anomaly Detection](https://mlsanity.com/paper/2310.18961/) (2023)
- [ChatIE: Zero-Shot Information Extraction via Chatting with ChatGPT](https://mlsanity.com/paper/2302.10205/) (2023)
- [GatorTron: A Large Clinical Language Model to Unlock Patient Information from Unstructured Electronic Health Records](https://mlsanity.com/paper/2203.03540/) (2022)
- [Structured Prediction as Translation between Augmented Natural Languages](https://mlsanity.com/paper/2101.05779/) (2021)
- [ZegCLIP: Towards Adapting CLIP for Zero-shot Semantic Segmentation](https://mlsanity.com/paper/2212.03588/) (2022)
- [InstructUIE: Multi-task Instruction Tuning for Unified Information Extraction](https://mlsanity.com/paper/2304.08085/) (2023)
- [GPT-NER: Named Entity Recognition via Large Language Models](https://mlsanity.com/paper/2304.10428/) (2023)
- [When MOE Meets LLMs: Parameter Efficient Fine-tuning for Multi-task Medical Applications](https://mlsanity.com/paper/2310.18339/) (2023)
- [Improving Named Entity Recognition by External Context Retrieving and Cooperative Learning](https://mlsanity.com/paper/2105.03654/) (2021)
- [Evaluating the Susceptibility of Pre-Trained Language Models via Handcrafted Adversarial Examples](https://mlsanity.com/paper/2209.02128/) (2022)
- [Empirical Study of Zero-Shot NER with ChatGPT](https://mlsanity.com/paper/2310.10035/) (2023)
- [Cross-City Matters: A Multimodal Remote Sensing Benchmark Dataset for Cross-City Semantic Segmentation using High-Resolution Domain Adaptation Networks](https://mlsanity.com/paper/2309.16499/) (2023)
- [Contrastive Grouping with Transformer for Referring Image Segmentation](https://mlsanity.com/paper/2309.01017/) (2023)
- [Towards Robust Referring Video Object Segmentation with Cyclic Relational Consensus](https://mlsanity.com/paper/2207.01203/) (2022)
- [Spectrum-guided Multi-granularity Referring Video Object Segmentation](https://mlsanity.com/paper/2307.13537/) (2023)
- [DeepStruct: Pretraining of Language Models for Structure Prediction](https://mlsanity.com/paper/2205.10475/) (2022)
- [Open-Vocabulary Semantic Segmentation with Decoupled One-Pass Network](https://mlsanity.com/paper/2304.01198/) (2023)
- [T-NER: An All-Round Python Library for Transformer-based Named Entity Recognition](https://mlsanity.com/paper/2209.12616/) (2022)
- [Pulse-PPG: An Open-Source Field-Trained PPG Foundation Model for Wearable Applications Across Lab and Field Settings](https://mlsanity.com/paper/2502.01108/) (2025)
- [Balancing Methods for Multi-label Text Classification with Long-Tailed Class Distribution](https://mlsanity.com/paper/2109.04712/) (2021)
- [MediCLIP: Adapting CLIP for Few-shot Medical Image Anomaly Detection](https://mlsanity.com/paper/2405.11315/) (2024)
- [Named Entity Recognition in Twitter: A Dataset and Analysis on Short-Term Temporal Shifts](https://mlsanity.com/paper/2210.03797/) (2022)
- [Naamapadam: A Large-Scale Named Entity Annotated Data for Indic Languages](https://mlsanity.com/paper/2212.10168/) (2022)
- [Pushing the Limits of ChatGPT on NLP Tasks](https://mlsanity.com/paper/2306.09719/) (2023)
- [Exploring Pre-trained Text-to-Video Diffusion Models for Referring Video Object Segmentation](https://mlsanity.com/paper/2403.12042/) (2024)
- [Cross-Modal Bidirectional Interaction Model for Referring Remote Sensing Image Segmentation](https://mlsanity.com/paper/2410.08613/) (2024)
- [Learning Opinion Summarizers by Selecting Informative Reviews](https://mlsanity.com/paper/2109.04325/) (2021)
- [Curriculum Point Prompting for Weakly-Supervised Referring Image Segmentation](https://mlsanity.com/paper/2404.11998/) (2024)
- [Better Feature Integration for Named Entity Recognition](https://mlsanity.com/paper/2104.05316/) (2021)
- [Optimising Equal Opportunity Fairness in Model Training](https://mlsanity.com/paper/2205.02393/) (2022)

## Abstract (arXiv)

Many NLP tasks such as tagging and machine reading comprehension are faced with the severe data imbalance issue: negative examples significantly outnumber positive examples, and the huge number of background examples (or easy-negative examples) overwhelms the training. The most commonly used cross entropy (CE) criteria is actually an accuracy-oriented objective, and thus creates a discrepancy between training and test: at training time, each training instance contributes equally to the objective function, while at test time F1 score concerns more about positive examples. In this paper, we propose to use dice loss in replacement of the standard cross-entropy objective for data-imbalanced NLP tasks. Dice loss is based on the Sorensen-Dice coefficient or Tversky index, which attaches similar importance to false positives and false negatives, and is more immune to the data-imbalance issue. To further alleviate the dominating influence from easy-negative examples in training, we propose to associate training examples with dynamically adjusted weights to deemphasize easy-negative examples.Theoretical analysis shows that this strategy narrows down the gap between the F1 score in evaluation and the dice loss in training. With the proposed training objective, we observe significant performance boost on a wide range of data imbalanced NLP tasks. Notably, we are able to achieve SOTA results on CTB5, CTB6 and UD1.4 for the part of speech tagging task; SOTA results on CoNLL03, OntoNotes5.0, MSRA and OntoNotes4.0 for the named entity recognition task; along with competitive results on the tasks of machine reading comprehension and paraphrase identification.