mlsanityData to 14 Oct 2026

Annotation Artifacts in Natural Language Inference Data

arXiv 1803.02324 ↗First posted 6 Mar 2018cs.CL cs.AIcited by 671

Not on the map. Why?

Cited by 40 in LLM reasoning, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Large-scale datasets for natural language inference are created by presenting crowd workers with a sentence (premise), and asking them to generate three new sentences (hypotheses) that it entails, contradicts, or is logically neutral with respect to. We show that, in a significant portion of such data, this protocol leaves clues that make it possible to identify the label by looking only at the hypothesis, without observing the premise. Specifically, we show that a simple text categorization model can correctly classify the hypothesis alone in about 67% of SNLI (Bowman et. al, 2015) and 53% of MultiNLI (Williams et. al, 2017). Our analysis reveals that specific linguistic phenomena such as negation and vagueness are highly correlated with certain inference classes. Our findings suggest that the success of natural language inference models to date has been overestimated, and that the task remains a hard open problem.

Key facts. Cited by 671 papers in the map (data through 14 Oct 2026). First posted on arXiv 6 Mar 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 153; 2022: 152; 2023: 125; 2024: 79; 2025: 60; 2026: 102.

671 papers.

Sort:

By area:

  1. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,418
  2. 2022Survey of Hallucination in Natural Language Generationcited by 2,075
  3. 2023Large Language Models are not Fair Evaluatorscited by 1,085
  4. 2021Are NLP Models really able to Solve Simple Math Word Problems?cited by 878
  5. 2021Did Aristotle Use a Laptop? A Question Answering Benchmark with Implicit Reasoning Strategiescited by 743
  6. 2023DataComp: In search of the next generation of multimodal datasetscited by 644
  7. 2022Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Modelcited by 559
  8. 2021Datasets: A Community Library for Natural Language Processingcited by 530
  9. 2021Just Train Twice: Improving Group Robustness without Training Group Informationcited by 523
  10. 2021Dynabench: Rethinking Benchmarking in NLPcited by 459
  11. 2022Unnatural Instructions: Tuning Language Models with (Almost) No Human Laborcited by 397
  12. 2021A Survey on Automated Fact-Checkingcited by 378
38 more
  1. 2021AI and the Everything in the Whole Wide World Benchmarkcited by 349
  2. 2021Understanding Dataset Difficulty with V-Usable Informationcited by 291
  3. 2023A Long Way to Go: Investigating Length Correlations in RLHFcited by 291
  4. 2021Symbolic Knowledge Distillation: from General Language Models to Commonsense Modelscited by 270
  5. 2022Prompting GPT-3 To Be Reliablecited by 260
  6. 2024MiniCheck: Efficient Fact-Checking of LLMs on Grounding Documentscited by 257
  7. 2023SugarCrepe: Fixing Hackable Benchmarks for Vision-Language Compositionalitycited by 243
  8. 2021Get Your Vitamin C! Robust Fact Verification with Contrastive Evidencecited by 225
  9. 2021Polyjuice: Generating Counterfactuals for Explaining, Evaluating, and Improving Modelscited by 222
  10. 2022Neural Theory-of-Mind? On the Limits of Social Intelligence in Large LMscited by 220
  11. 2022Leveraging Large Language Models for Multiple Choice Question Answeringcited by 203
  12. 2022WANLI: Worker and AI Collaboration for Natural Language Inference Dataset Creationcited by 202
  13. 2024Are We Done with MMLU?cited by 194
  14. 2021Causal Inference in Natural Language Processing: Estimation, Prediction, Interpretation and Beyondcited by 178
  15. 2021Masked Language Modeling and the Distributional Hypothesis: Order Word Matters Pre-training for Littlecited by 177
  16. 2021Hurdles to Progress in Long-form Question Answeringcited by 167
  17. 2023GPT-NER: Named Entity Recognition via Large Language Modelscited by 160
  18. 2023Should ChatGPT be Biased? Challenges and Risks of Bias in Large Language Modelscited by 148
  19. 2022On Feature Learning in the Presence of Spurious Correlationscited by 145
  20. 2022Impact of Pretraining Term Frequencies on Few-Shot Reasoningcited by 142
  21. 2022The Debate Over Understanding in AI's Large Language Modelscited by 142
  22. 2023Inspecting and Editing Knowledge Representations in Language Modelscited by 141
  23. 2021The Benchmark Lotterycited by 136
  24. 2021VALSE: A Task-Independent Benchmark for Vision and Language Models Centered on Linguistic Phenomenacited by 135
  25. 2022Revealing Single Frame Bias for Video-and-Language Learningcited by 133
  26. 2021Teach Me to Explain: A Review of Datasets for Explainable Natural Language Processingcited by 126
  27. 2021Reframing Human-AI Collaboration for Generating Free-Text Explanationscited by 125
  28. 2023WiCE: Real-World Entailment for Claims in Wikipediacited by 120
  29. 2021HiTab: A Hierarchical Table Dataset for Question Answering and Natural Language Generationcited by 118
  30. 2021KLUE: Korean Language Understanding Evaluationcited by 114
  31. 2022NumGLUE: A Suite of Fundamental yet Challenging Mathematical Reasoning Taskscited by 112
  32. 2021Robustness Gym: Unifying the NLP Evaluation Landscapecited by 108
  33. 2021Measure and Improve Robustness in NLP Models: A Surveycited by 107
  34. 2022Unveiling Transformers with LEGO: a synthetic reasoning taskcited by 106
  35. 2024Relying on the Unreliable: The Impact of Language Models' Reluctance to Express Uncertaintycited by 106
  36. 2023SCOTT: Self-Consistent Chain-of-Thought Distillationcited by 105
  37. 2021UNICORN on RAINBOW: A Universal Commonsense Reasoning Model on a New Multitask Benchmarkcited by 98
  38. 2022State-of-the-art generalisation research in NLP: A taxonomy and reviewcited by 98

50 most cited of 671 All 671 (JSON)