mlsanityData to 7 Oct 2026

PECO: Examining Single Sentence Label Leakage in Natural Language Inference Datasets through Progressive Evaluation of Cluster Outliers

arXiv 2112.09237 ↗First posted 16 Dec 2021cs.CLcited by 8

Not on the map. Why?

Abstract (arXiv)

Building natural language inference (NLI) benchmarks that are both challenging for modern techniques, and free from shortcut biases is difficult. Chief among these biases is "single sentence label leakage," where annotator-introduced spurious correlations yield datasets where the logical relation between (premise, hypothesis) pairs can be accurately predicted from only a single sentence, something that should in principle be impossible. We demonstrate that despite efforts to reduce this leakage, it persists in modern datasets that have been introduced since its 2018 discovery. To enable future amelioration efforts, introduce a novel model-driven technique, the progressive evaluation of cluster outliers (PECO) which enables both the objective measurement of leakage, and the automated detection of subpopulations in the data which maximally exhibit it.

Key facts. Cited by 8 papers in the map (data through 7 Oct 2026). Builds on 40 of its 56 references that are in the map. First posted on arXiv 16 Dec 2021.

Builds on

40 of 56 references.

References · 40

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  4. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
  5. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,686
  6. 2017Axiomatic Attribution for Deep Networkscited by 3,350
  7. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
  8. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
  9. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  10. 2018FEVER: a large-scale dataset for Fact Extraction and VERificationcited by 1,362
  11. 2018XNLI: Evaluating Cross-lingual Sentence Representationscited by 913
  12. 2019Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inferencecited by 864
28 more
  1. 2019Adversarial NLI: A New Benchmark for Natural Language Understandingcited by 798
  2. 2018Annotation Artifacts in Natural Language Inference Datacited by 656
  3. 2020KILT: a Benchmark for Knowledge Intensive Language Taskscited by 571
  4. 2021ByT5: Towards a token-free future with pre-trained byte-to-byte modelscited by 504
  5. 2020Dataset Cartography: Mapping and Diagnosing Datasets with Training Dynamicscited by 423
  6. 2018Hypothesis Only Baselines in Natural Language Inferencecited by 291
  7. 2020CLUE: A Chinese Language Understanding Evaluation Benchmarkcited by 277
  8. 2021What Will it Take to Fix Benchmarking in Natural Language Understanding?cited by 200
  9. 2019Are We Modeling the Task or the Annotator? An Investigation of Annotator Bias in Natural Language Understanding Datasetscited by 173
  10. 2020An Empirical Study on Robustness to Spurious Correlations using Pre-trained Language Modelscited by 133
  11. 2021Entailment as Few-Shot Learnercited by 122
  12. 2018Combining Fact Extraction and Verification with Neural Semantic Matching Networkscited by 120
  13. 2020InfoBERT: Improving Robustness of Language Models from An Information Theoretic Perspectivecited by 92
  14. 2021Competency Problems: On Finding and Removing Artifacts in Language Datacited by 91
  15. 2020OCNLI: Original Chinese Natural Language Inferencecited by 86
  16. 2020Syntactic Data Augmentation Increases Robustness to Inference Heuristicscited by 85
  17. 2021Generalization in NLI: Ways (Not) To Go Beyond Simple Heuristicscited by 67
  18. 2020Explaining The Efficacy of Counterfactually Augmented Datacited by 64
  19. 2019Don't Take the Premise for Granted: Mitigating Artifacts in Natural Language Inferencecited by 48
  20. 2020Towards Robustifying NLI Models Against Lexical Dataset Biasescited by 34
  21. 2020Counterfactually-Augmented SNLI Training Data Does Not Yield Better Generalization Than Unaugmented Datacited by 27
  22. 2021NeuralLog: Natural Language Inference with Joint Neural and Logical Reasoningcited by 24
  23. 2021End-to-End Spoken Language Understanding for Generalized Voice Assistantscited by 17
  24. 2020Semantic Complexity in End-to-End Spoken Language Understandingcited by 10
  25. 2020Improving Robustness by Augmenting Training Sentences with Predicate-Argument Structurescited by 10
  26. 2019Selection Bias Explorations and Debias Methods for Natural Language Sentence Matching Datasetscited by 8
  27. 2021Unsupervised Natural Language Inference Using PHL Triplet Generationcited by 6
  28. 2021Counterfactual Maximum Likelihood Estimation for Training Deep Networkscited by 5

Built on by

Cited by year

Citing papers per year: 2022: 1; 2023: 2; 2024: 4; 2025: 1.

8 papers.

  1. 2023Automatically Correcting Large Language Models: Surveying the landscape of diverse self-correction strategiescited by 236
  2. 2022Shortcut Learning of Large Language Models in Natural Language Understandingcited by 83
  3. 2025Which of These Best Describes Multiple Choice Evaluation with LLMs? A) Forced B) Flawed C) Fixable D) All of the Abovecited by 51
  4. 2024Benchmarks as Microscopes: A Call for Model Metrologycited by 33
  5. 2024Who Evaluates the Evaluations? Objectively Scoring Text-to-Image Prompt Coherence Metrics with T2IScoreScore (TS2)cited by 17
  6. 2023Multilingual Conceptual Coverage in Text-to-Image Modelscited by 9
  7. 2024From Form(s) to Meaning: Probing the Semantic Depths of Language Models Using Multisense Consistencycited by 7
  8. 2024Lost in Translation? Translation Errors and Challenges for Fair Assessment of Text-to-Image Models on Multilingual Conceptscited by 5