mlsanityData to 7 Oct 2026

How Does Data Corruption Affect Natural Language Understanding Models? A Study on GLUE datasets

arXiv 2201.04467 ↗First posted 12 Jan 2022cs.CLcited by 0

On the map: LLM reasoning → Reasoning benchmarks and evaluation

Abstract (arXiv)

A central question in natural language understanding (NLU) research is whether high performance demonstrates the models' strong reasoning capabilities. We present an extensive series of controlled experiments where pre-trained language models are exposed to data that have undergone specific corruption transformations. These involve removing instances of specific word classes and often lead to non-sensical sentences. Our results show that performance remains high on most GLUE tasks when the models are fine-tuned or tested on corrupted data, suggesting that they leverage other cues for prediction even in non-sensical contexts. Our proposed data transformations can be used to assess the extent to which a specific dataset constitutes a proper testbed for evaluating models' language understanding capabilities.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 27 of its 38 references that are in the map. First posted on arXiv 12 Jan 2022.

Builds on

27 of 38 references.

References · 27

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  3. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  4. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  5. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
  6. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,718
  7. 2017SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluationcited by 998
  8. 2018Neural Network Acceptability Judgmentscited by 952
  9. 2019Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inferencecited by 864
  10. 2019Adversarial NLI: A New Benchmark for Natural Language Understandingcited by 798
  11. 2018Annotation Artifacts in Natural Language Inference Datacited by 656
  12. 2018SWAG: A Large-Scale Adversarial Dataset for Grounded Commonsense Inferencecited by 333
15 more
  1. 2019What BERT is not: Lessons from a new suite of psycholinguistic diagnostics for language modelscited by 317
  2. 2018Hypothesis Only Baselines in Natural Language Inferencecited by 291
  3. 2013Crowdsourcing a Word-Emotion Association Lexiconcited by 193
  4. 2021Masked Language Modeling and the Distributional Hypothesis: Order Word Matters Pre-training for Littlecited by 176
  5. 2019Neural Language Models as Psycholinguistic Subjects: Representations of Syntactic Statecited by 137
  6. 2018Breaking NLI Systems with Sentences that Require Simple Lexical Inferencescited by 134
  7. 2020Out of Order: How Important Is The Sequential Order of Words in a Sentence in Natural Language Understanding Tasks?cited by 72
  8. 2018Performance Impact Caused by Hidden Bias of Training Data for Recognizing Textual Entailmentcited by 58
  9. 2021The paradox of the compositionality of natural language: a neural machine translation case studycited by 58
  10. 2019Assessing the Benchmarking Capacity of Machine Reading Comprehension Datasetscited by 46
  11. 2017The Effect of Different Writing Tasks on Linguistic Style: A Case Study of the ROC Story Cloze Taskcited by 39
  12. 2019What does BERT Learn from Multiple-Choice Reading Comprehension Datasets?cited by 30
  13. 2018Testing the Generalization Power of Neural Network Models Across NLI Benchmarkscited by 15
  14. 2021Shaking Syntactic Trees on the Sesame Street: Multilingual Probing with Controllable Perturbationscited by 10
  15. 2021NLI Data Sanity Check: Assessing the Effect of Data Corruption on Model Performancecited by 4

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026SynthSentry: Detecting Synthetic Data Contamination in Language Model Training Data
  2. 2022Towards Faithful Model Explanation in NLP: A Survey
  3. 2023Can NLP Models Correctly Reason Over Contexts that Break the Common Assumptions?
  4. 2025A Causal Lens for Evaluating Faithfulness Metrics
  5. 2025On the Notion that Language Models Reason
  6. 2024Unveiling the Flaws: Exploring Imperfections in Synthetic Data and Mitigation Strategies for Large Language Models
  7. 2024LLMs' Understanding of Natural Language Revealed
  8. 2024How often are errors in natural language reasoning due to paraphrastic variability?