mlsanityData to 7 Oct 2026

Capturing Label Distribution: A Case Study in NLI

arXiv 2102.06859 ↗First posted 13 Feb 2021cs.CLcited by 6

Not on the map. Why?

Abstract (arXiv)

We study estimating inherent human disagreement (annotation label distribution) in natural language inference task. Post-hoc smoothing of the predicted label distribution to match the expected label entropy is very effective. Such simple manipulation can reduce KL divergence by almost half, yet will not improve majority label prediction accuracy or learn label distributions. To this end, we introduce a small amount of examples with multiple references into training. We depart from the standard practice of collecting a single reference per each training example, and find that collecting multiple references can achieve better accuracy under the fixed annotation budget. Lastly, we provide rich analyses comparing these two methods for improving label distribution estimation.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Builds on 25 of its 39 references that are in the map. First posted on arXiv 13 Feb 2021.

Builds on

25 of 39 references.

References · 25

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  4. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,451
  5. 2017On Calibration of Modern Neural Networkscited by 4,860
  6. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  7. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
  8. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
  9. 2015A large annotated corpus for learning natural language inferencecited by 1,837
  10. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,718
  11. 2020Dataset Cartography: Mapping and Diagnosing Datasets with Training Dynamicscited by 423
  12. 2017On Fairness and Calibrationcited by 406
13 more
  1. 2020Calibration of Pre-trained Transformerscited by 307
  2. 2019On Mixup Training: Improved Calibration and Predictive Uncertainty for Deep Neural Networkscited by 304
  3. 2019Abductive Commonsense Reasoningcited by 300
  4. 2020Selective Question Answering under Domain Shiftcited by 228
  5. 2018To Trust Or Not To Trust A Classifiercited by 205
  6. 2019Show Your Work: Improved Reporting of Experimental Resultscited by 164
  7. 2019Unifying Human and Statistical Evaluation for Natural Language Generationcited by 127
  8. 2018Analyzing Uncertainty in Neural Machine Translationcited by 126
  9. 2017Learning how to Active Learn: A Deep Reinforcement Learning Approachcited by 79
  10. 2015Posterior calibration and exploratory analysis for natural language processing modelscited by 75
  11. 2016Ordinal Common-sense Inferencecited by 35
  12. 2019Uncertain Natural Language Inferencecited by 35
  13. 2020Learning a Cost-Effective Annotation Policy for Question Answeringcited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 2; 2022: 0; 2023: 2; 2024: 0; 2025: 1; 2026: 1.

6 papers.

  1. 2021Distributed NLI: Learning to Predict Human Opinion Distributions for Language Reasoningcited by 31
  2. 2023POUF: Prompt-oriented unsupervised fine-tuning for large pre-trained modelscited by 29
  3. 2021Learning with Different Amounts of Annotation: From Zero to Many Labelscited by 28
  4. 2023No Strong Feelings One Way or Another: Re-operationalizing Neutrality in Natural Language Inferencecited by 4
  5. 2025Giving AI Personalities Leads to More Human-Like Reasoningcited by 3
  6. 2026Metric-Dependent Annotation Saturation for Learning from Label Distributionscited by 0