mlsanityData to 7 Oct 2026

Evaluating Paraphrastic Robustness in Textual Entailment Models

arXiv 2306.16722 ↗First posted 29 Jun 2023cs.CL cs.AIcited by 7

Not on the map. Why?

Abstract (arXiv)

We present PaRTE, a collection of 1,126 pairs of Recognizing Textual Entailment (RTE) examples to evaluate whether models are robust to paraphrasing. We posit that if RTE models understand language, their predictions should be consistent across inputs that share the same meaning. We use the evaluation set to determine if RTE models' predictions change when examples are paraphrased. In our experiments, contemporary models change their predictions on 8-16% of paraphrased examples, indicating that there is still room for improvement.

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Builds on 24 of its 57 references that are in the map. First posted on arXiv 29 Jun 2023.

Builds on

24 of 57 references.

References · 24

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  3. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  4. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
  5. 2017Adversarial Examples for Evaluating Reading Comprehension Systemscited by 568
  6. 2019PAWS: Paraphrase Adversaries from Word Scramblingcited by 395
  7. 2017Synthetic and Natural Noise Both Break Neural Machine Translationcited by 304
  8. 2019Benchmarking Zero-shot Text Classification: Datasets, Evaluation and Entailment Approachcited by 299
  9. 2018Hypothesis Only Baselines in Natural Language Inferencecited by 291
  10. 2018Adversarial Example Generation with Syntactically Controlled Paraphrase Networkscited by 282
  11. 2018Stress Test Evaluation for Natural Language Inferencecited by 211
  12. 2022WANLI: Worker and AI Collaboration for Natural Language Inference Dataset Creationcited by 200
12 more
  1. 2017Generating Natural Adversarial Examplescited by 182
  2. 2022Ask Me Anything: A simple strategy for prompting language modelscited by 155
  3. 2020Are Natural Language Inference Models IMPPRESsive? Learning IMPlicature and PRESuppositioncited by 76
  4. 2019Perturbation Sensitivity Analysis to Detect Unintended Model Biasescited by 74
  5. 2019Probing Natural Language Inference Models through Semantic Fragmentscited by 64
  6. 2019Can neural networks understand monotonicity reasoning?cited by 45
  7. 2022CONDAQA: A Contrastive Reading Comprehension Dataset for Reasoning about Negationcited by 38
  8. 2019HELP: A Dataset for Identifying Shortcomings of Neural Models in Monotonicity Reasoningcited by 35
  9. 2022Falsesum: Generating Document-level NLI Examples for Recognizing Factual Inconsistency in Summarizationcited by 34
  10. 2021Figurative Language in Recognizing Textual Entailmentcited by 26
  11. 2021NOPE: A Corpus of Naturally-Occurring Presuppositions in Englishcited by 23
  12. 2021Exploring Transitivity in Neural NLI Models through Veridicalitycited by 15

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 3; 2025: 3; 2026: 1.

7 papers.

  1. 2024Fine-Grained Natural Language Inference Based Faithfulness Evaluation for Diverse Summarisation Taskscited by 23
  2. 2025NLI under the Microscope: What Atomic Hypothesis Decomposition Revealscited by 9
  3. 2025MuSciClaims: Multimodal Scientific Claim Verificationcited by 8
  4. 2024How often are errors in natural language reasoning due to paraphrastic variability?cited by 5
  5. 2024Simpler becomes Harder: Do LLMs Exhibit a Coherent Behavior on Simplified Corpora?cited by 1
  6. 2026Same Patient, Different Words, Different Diagnosis? Evaluating Semantic Stability in Clinical LLMscited by 1
  7. 2025MERGE: Minimal Expression-Replacement GEneralization Test for Natural Language Inferencecited by 0