mlsanityData to 7 Oct 2026

Breaking NLI Systems with Sentences that Require Simple Lexical Inferences

arXiv 1805.02266 ↗First posted 6 May 2018cs.CLcited by 134

Not on the map. Why?

Abstract (arXiv)

We create a new NLI test set that shows the deficiency of state-of-the-art models in inferences that require lexical and world knowledge. The new examples are simpler than the SNLI test set, containing sentences that differ by at most one word from sentences in the training set. Yet, the performance on the new test set is substantially worse across systems trained on SNLI, demonstrating that these systems are limited in their generalization ability, failing to capture many simple inferences.

Key facts. Cited by 134 papers in the map (data through 7 Oct 2026). First posted on arXiv 6 May 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 42; 2022: 43; 2023: 28; 2024: 11; 2025: 7; 2026: 3.

134 papers.

Sort:

By area:

  1. 2021Dynabench: Rethinking Benchmarking in NLPcited by 447
  2. 2021A Survey of Data Augmentation Approaches for NLPcited by 408
  3. 2021Polyjuice: Generating Counterfactuals for Explaining, Evaluating, and Improving Modelscited by 219
  4. 2022WANLI: Worker and AI Collaboration for Natural Language Inference Dataset Creationcited by 200
  5. 2023Clever Hans or Neural Theory of Mind? Stress Testing Social Reasoning in Large Language Modelscited by 161
  6. 2021KLUE: Korean Language Understanding Evaluationcited by 113
  7. 2021Robustness Gym: Unifying the NLP Evaluation Landscapecited by 107
  8. 2023Generating Benchmarks for Factuality Evaluation of Language Modelscited by 103
  9. 2022State-of-the-art generalisation research in NLP: A taxonomy and reviewcited by 97
  10. 2023T-Eval: Evaluating the Tool Utilization Capability of Large Language Models Step by Stepcited by 91
  11. 2022FLUTE: Figurative Language Understanding through Textual Explanationscited by 78
  12. 2024Artifacts or Abduction: How Do LLMs Answer Multiple-Choice Questions Without the Question?cited by 68
38 more
  1. 2022Investigating Selective Prediction Approaches Across Several Tasks in IID, OOD, and Adversarial Settingscited by 66
  2. 2022Linear Connectivity Reveals Generalization Strategiescited by 55
  3. 2022Generating Data to Mitigate Spurious Correlations in Natural Language Inference Datasetscited by 54
  4. 2022Predicting Out-of-Distribution Error with the Projection Normcited by 50
  5. 2021COM2SENSE: A Commonsense Reasoning Benchmark with Complementary Sentencescited by 40
  6. 2021Back to Square One: Artifact Detection, Training and Commonsense Disentanglement in the Winograd Schemacited by 32
  7. 2024LLMs for Generating and Evaluating Counterfactuals: A Comprehensive Studycited by 30
  8. 2022DISCO: Distilling Counterfactuals with Large Language Modelscited by 29
  9. 2021Evaluating Gender Bias in Natural Language Inferencecited by 27
  10. 2021Influence Tuning: Demoting Spurious Correlations via Instance Attribution and Instance-Driven Updatescited by 27
  11. 2021Figurative Language in Recognizing Textual Entailmentcited by 26
  12. 2021NoiseQA: Challenge Set Evaluation for User-Centric Question Answeringcited by 25
  13. 2022Textual Entailment for Event Argument Extraction: Zero- and Few-Shot with Multi-Source Learningcited by 25
  14. 2021NeuralLog: Natural Language Inference with Joint Neural and Logical Reasoningcited by 24
  15. 2021Analyzing Dynamic Adversarial Training Data in the Limitcited by 24
  16. 2021IndoNLI: A Natural Language Inference Dataset for Indonesiancited by 24
  17. 2023CREST: A Joint Framework for Rationalization and Counterfactual Text Generationcited by 24
  18. 2021Avoiding Inference Heuristics in Few-shot Prompt-based Finetuningcited by 23
  19. 2022Generalized but not Robust? Comparing the Effects of Data Modification Methods on Out-of-Domain Generalization and Adversarial Robustnesscited by 23
  20. 2022CORE: A Retrieve-then-Edit Framework for Counterfactual Data Generationcited by 23
  21. 2022NeuroCounterfactuals: Beyond Minimal-Edit Counterfactuals for Richer Data Augmentationcited by 23
  22. 2021Adversarial Examples for Evaluating Math Word Problem Solverscited by 20
  23. 2021ReaSCAN: Compositional Reasoning in Language Groundingcited by 19
  24. 2022On the Limitations of Dataset Balancing: The Lost Battle Against Spurious Correlationscited by 19
  25. 2021Exploring Transitivity in Neural NLI Models through Veridicalitycited by 15
  26. 2021SciCo: Hierarchical Cross-Document Coreference for Scientific Conceptscited by 14
  27. 2022Compositional Evaluation on Japanese Textual Entailment and Similaritycited by 14
  28. 2022Visual Comparison of Language Model Adaptationcited by 14
  29. 2024Semantic Sensitivities and Inconsistent Predictions: Measuring the Fragility of NLI Modelscited by 14
  30. 2024Is Your Large Language Model Knowledgeable or a Choices-Only Cheater?cited by 12
  31. 2022Metaethical Perspectives on 'Benchmarking' AI Ethicscited by 11
  32. 2023Understanding and Mitigating Spurious Correlations in Text Classification with Neighborhood Analysiscited by 11
  33. 2024Resolving Word Vagueness with Scenario-guided Adapter for Natural Language Inferencecited by 11
  34. 2021Mitigating the Position Bias of Transformer Models in Passage Re-Rankingcited by 10
  35. 2021Probing Pre-Trained Language Models for Disease Knowledgecited by 10
  36. 2022AGRO: Adversarial Discovery of Error-prone groups for Robust Optimizationcited by 10
  37. 2022Feature-Level Debiased Natural Language Understandingcited by 10
  38. 2021Language Models for Lexical Inference in Contextcited by 9

50 most cited of 134 All 134 (JSON)