mlsanityData to 7 Oct 2026

Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inference

arXiv 1902.01007 ↗First posted 4 Feb 2019cs.CLcited by 864

Not on the map. Why?

Cited by 59 in LLM reasoning, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

A machine learning system can score well on a given test set by relying on heuristics that are effective for frequent example types but break down in more challenging cases. We study this issue within natural language inference (NLI), the task of determining whether one sentence entails another. We hypothesize that statistical NLI models may adopt three fallible syntactic heuristics: the lexical overlap heuristic, the subsequence heuristic, and the constituent heuristic. To determine whether models have adopted these heuristics, we introduce a controlled evaluation set called HANS (Heuristic Analysis for NLI Systems), which contains many examples where the heuristics fail. We find that models trained on MNLI, including BERT, a state-of-the-art model, perform very poorly on HANS, suggesting that they have indeed adopted these heuristics. We conclude that there is substantial room for improvement in NLI systems, and that the HANS dataset can motivate and measure progress in this area

Key facts. Cited by 864 papers in the map (data through 7 Oct 2026). First posted on arXiv 4 Feb 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 181; 2022: 192; 2023: 157; 2024: 108; 2025: 99; 2026: 127.

864 papers.

Sort:

By area:

  1. 2023Large Language Models are not Fair Evaluatorscited by 1,050
  2. 2021Are NLP Models really able to Solve Simple Math Word Problems?cited by 861
  3. 2022Discovering Language Model Behaviors with Model-Written Evaluationscited by 805
  4. 2023Unifying Large Language Models and Knowledge Graphs: A Roadmapcited by 663
  5. 2023A Comprehensive Overview of Large Language Modelscited by 639
  6. 2022Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Modelcited by 557
  7. 2021Just Train Twice: Improving Group Robustness without Training Group Informationcited by 522
  8. 2021Dynabench: Rethinking Benchmarking in NLPcited by 447
  9. 2022Last Layer Re-Training is Sufficient for Robustness to Spurious Correlationscited by 379
  10. 2021A Survey on Automated Fact-Checkingcited by 376
  11. 2021Measuring and Improving Consistency in Pretrained Language Modelscited by 361
  12. 2021Do Prompt-Based Models Really Understand the Meaning of their Prompts?cited by 324
38 more
  1. 2023Emergent Linear Representations in World Models of Self-Supervised Sequence Modelscited by 313
  2. 2021Factual Probing Is [MASK]: Learning vs. Learning to Recallcited by 291
  3. 2022Prompting GPT-3 To Be Reliablecited by 258
  4. 2021Get Your Vitamin C! Robust Fact Verification with Contrastive Evidencecited by 222
  5. 2021Polyjuice: Generating Counterfactuals for Explaining, Evaluating, and Improving Modelscited by 219
  6. 2022Exploring Length Generalization in Large Language Modelscited by 204
  7. 2021What Will it Take to Fix Benchmarking in Natural Language Understanding?cited by 200
  8. 2022WANLI: Worker and AI Collaboration for Natural Language Inference Dataset Creationcited by 200
  9. 2021Muppet: Massive Multi-task Representations with Pre-Finetuningcited by 192
  10. 2021Causal Inference in Natural Language Processing: Estimation, Prediction, Interpretation and Beyondcited by 177
  11. 2021Masked Language Modeling and the Distributional Hypothesis: Order Word Matters Pre-training for Littlecited by 176
  12. 2022GreaseLM: Graph REASoning Enhanced Language Models for Question Answeringcited by 170
  13. 2021Post-hoc Interpretability for Neural NLP: A Surveycited by 155
  14. 2022Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Textcited by 154
  15. 2022Instruction Induction: From Few Examples to Natural Language Task Descriptionscited by 152
  16. 2022On Feature Learning in the Presence of Spurious Correlationscited by 145
  17. 2023Should ChatGPT be Biased? Challenges and Risks of Bias in Large Language Modelscited by 144
  18. 2022Impact of Pretraining Term Frequencies on Few-Shot Reasoningcited by 142
  19. 2022The Debate Over Understanding in AI's Large Language Modelscited by 142
  20. 2023Few-shot Fine-tuning vs. In-context Learning: A Fair Comparison and Evaluationcited by 134
  21. 2022Towards Faithful Model Explanation in NLP: A Surveycited by 129
  22. 2023Similarity of Neural Network Models: A Survey of Functional and Representational Measurescited by 128
  23. 2023WiCE: Real-World Entailment for Claims in Wikipediacited by 118
  24. 2021Gender Bias in Machine Translationcited by 116
  25. 2021A Survey on Data Augmentation for Text Classificationcited by 111
  26. 2021Robustness Gym: Unifying the NLP Evaluation Landscapecited by 107
  27. 2021Knowledgeable or Educated Guess? Revisiting Language Models as Knowledge Basescited by 107
  28. 2021Measure and Improve Robustness in NLP Models: A Surveycited by 107
  29. 2022Unveiling Transformers with LEGO: a synthetic reasoning taskcited by 105
  30. 2021Abstraction and Analogy-Making in Artificial Intelligencecited by 103
  31. 2022Plex: Towards Reliability using Pretrained Large Model Extensionscited by 102
  32. 2025Measuring what Matters: Construct Validity in Large Language Model Benchmarkscited by 102
  33. 2021An Empirical Survey of Data Augmentation for Limited Data Learning in NLPcited by 101
  34. 2022State-of-the-art generalisation research in NLP: A taxonomy and reviewcited by 97
  35. 2023Revisiting Out-of-distribution Robustness in NLP: Benchmark, Analysis, and LLMs Evaluationscited by 96
  36. 2022On the Paradox of Learning to Reason from Datacited by 93
  37. 2023Language Model Behavior: A Comprehensive Surveycited by 92
  38. 2021QA Dataset Explosion: A Taxonomy of NLP Resources for Question Answering and Reading Comprehensioncited by 91

50 most cited of 864 All 864 (JSON)