mlsanityData to 14 Oct 2026

MENLI: Robust Evaluation Metrics from Natural Language Inference

arXiv 2208.07316 ↗First posted 15 Aug 2022cs.CL cs.CR cs.LGcited by 18

Not on the map. Why?

Abstract (arXiv)

Recently proposed BERT-based evaluation metrics for text generation perform well on standard benchmarks but are vulnerable to adversarial attacks, e.g., relating to information correctness. We argue that this stems (in part) from the fact that they are models of semantic similarity. In contrast, we develop evaluation metrics based on Natural Language Inference (NLI), which we deem a more appropriate modeling. We design a preference-based adversarial attack framework and show that our NLI based metrics are much more robust to the attacks than the recent BERT-based metrics. On standard benchmarks, our NLI based metrics outperform existing summarization metrics, but perform below SOTA MT metrics. However, when combining existing metrics with our NLI metrics, we obtain both higher adversarial robustness (15%-30%) and higher quality metrics as measured on standard benchmarks (+5% to 30%).

Key facts. Cited by 18 papers in the map (data through 14 Oct 2026). Builds on 58 of its 96 references that are in the map. First posted on arXiv 15 Aug 2022.

Builds on

58 of 96 references.

References · 58

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  3. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,512
  4. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,935
  5. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,682
  6. 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,181
  7. 2022Survey of Hallucination in Natural Language Generationcited by 2,075
  8. 2015A large annotated corpus for learning natural language inferencecited by 1,847
  9. 2021DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharingcited by 1,254
  10. 2020BLEURT: Learning Robust Metrics for Text Generationcited by 1,203
  11. 2015Teaching Machines to Read and Comprehendcited by 1,049
  12. 2020COMET: A Neural Framework for MT Evaluationcited by 1,033
38 more
  1. 2020Beyond Accuracy: Behavioral Testing of NLP models with CheckListcited by 1,006
  2. 2018XNLI: Evaluating Cross-lingual Sentence Representationscited by 915
  3. 2019Adversarial NLI: A New Benchmark for Natural Language Understandingcited by 807
  4. 2021BARTScore: Evaluating Generated Text as Text Generationcited by 722
  5. 2020SummEval: Re-evaluating Summarization Evaluationcited by 674
  6. 2018Annotation Artifacts in Natural Language Inference Datacited by 671
  7. 2019Evaluating the Factual Consistency of Abstractive Text Summarizationcited by 584
  8. 2019MoverScore: Text Generation Evaluating with Contextualized Embeddings and Earth Mover Distancecited by 418
  9. 2021SummaC: Re-Visiting NLI-based Models for Inconsistency Detection in Summarizationcited by 407
  10. 2019PAWS: Paraphrase Adversaries from Word Scramblingcited by 402
  11. 2021Experts, Errors, and Context: A Large-Scale Study of Human Evaluation for Machine Translationcited by 338
  12. 2018Hypothesis Only Baselines in Natural Language Inferencecited by 295
  13. 2019PAWS-X: A Cross-lingual Adversarial Dataset for Paraphrase Identificationcited by 264
  14. 2019Probing Neural Network Comprehension of Natural Language Argumentscited by 214
  15. 2016Enhanced LSTM for Natural Language Inferencecited by 200
  16. 2020Tangled up in BLEU: Reevaluating the Evaluation of Automatic Machine Translation Evaluation Metricscited by 158
  17. 2020Re-evaluating Evaluation in Text Summarizationcited by 145
  18. 2018Combining Fact Extraction and Verification with Neural Semantic Matching Networkscited by 120
  19. 2020Automatic Machine Translation Evaluation in Many Languages via Zero-Shot Paraphrasingcited by 115
  20. 2020Towards Question-Answering as an Automatic Metric for Evaluating the Content Quality of a Summarycited by 88
  21. 2020Mind the Trade-off: Debiasing NLU Models without Degrading the In-distribution Performancecited by 85
  22. 2021DocNLI: A Large-scale Dataset for Document-level Natural Language Inferencecited by 75
  23. 2019Cross-Lingual Alignment of Contextual Word Embeddings, with Applications to Zero-shot Dependency Parsingcited by 70
  24. 2020SUPERT: Towards New Frontiers in Unsupervised Evaluation Metrics for Multi-Document Summarizationcited by 69
  25. 2020TransQuest: Translation Quality Estimation with Cross-lingual Transformerscited by 59
  26. 2020Unbabel's Participation in the WMT20 Metrics Shared Taskcited by 56
  27. 2020Pretraining with Contrastive Sentence Objectives Improves Discourse Performance of Language Modelscited by 50
  28. 2021Perturbation CheckLists for Evaluating NLG Evaluation Metricscited by 50
  29. 2020Evaluating Semantic Accuracy of Data-to-Text Generation with Natural Language Inferencecited by 43
  30. 2016How Grammatical is Character-level Neural Machine Translation? Assessing MT Quality with Contrastive Translation Pairscited by 39
  31. 2019Crowdsourcing Lightweight Pyramids for Manual Summary Evaluationcited by 39
  32. 2021Automatic Text Evaluation through the Lens of Wasserstein Barycenterscited by 38
  33. 2020On the Limitations of Cross-lingual Encoders as Exposed by Reference-Free Machine Translation Evaluationcited by 36
  34. 2022DiscoScore: Evaluating Text Generation with BERT and Discourse Coherencecited by 36
  35. 2020Towards Robustifying NLI Models Against Lexical Dataset Biasescited by 34
  36. 2022Falsesum: Generating Document-level NLI Examples for Recognizing Factual Inconsistency in Summarizationcited by 34
  37. 2019On Adversarial Removal of Hypothesis-only Bias in Natural Language Inferencecited by 29
  38. 2022UniTE: Unified Translation Evaluationcited by 28

50 most cited of 58 All 58 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 2; 2023: 5; 2024: 1; 2025: 6; 2026: 4.

18 papers.

Sort:

By area:

  1. 2024TrustScore: Reference-Free Evaluation of LLM Response Trustworthinesscited by 19
  2. 2023Towards Explainable Evaluation Metrics for Machine Translationcited by 15
  3. 2023With a Little Push, NLI Models can Robustly and Efficiently Predict Faithfulnesscited by 13
  4. 2022Layer or Representation Space: What makes BERT-based Evaluation Metrics Robust?cited by 10
  5. 2025The Interspeech 2025 Speech Accessibility Project Challengecited by 9
  6. 2023Cross-lingual Cross-temporal Summarization: Dataset, Models, Evaluationcited by 8
  7. 2025Aligning ASR Evaluation with Human and LLM Judgments: Intelligibility Metrics Using Phonetic, Semantic, and NLI Approachescited by 7
  8. 2022Transformers Go for the LOLs: Generating (Humourous) Titles from Scientific Abstracts End-to-Endcited by 6
  9. 2025Evaluating GRPO and DPO for Faithful Chain-of-Thought Reasoning in LLMscited by 6
  10. 2023BLEU Meets COMET: Combining Lexical and Neural Metrics Towards Robust Machine Translation Evaluationcited by 5
  11. 2023Can GPT models Follow Human Summarization Guidelines? A Study for Targeted Communication Goalscited by 4
  12. 2025CROC: Evaluating and Training T2I Metrics with Pseudo- and Human-Labeled Contrastive Robustness Checkscited by 2
6 more
  1. 2026Same Patient, Different Words, Different Diagnosis? Evaluating Semantic Stability in Clinical LLMscited by 2
  2. 2025LaViPlan : Language-Guided Visual Path Planning with RLVRcited by 1
  3. 2025Music Recommendation with Large Language Models: Challenges, Opportunities, and Evaluationcited by 1
  4. 2026A Sentence Relation-Based Approach to Sanitizing Malicious Instructionscited by 0
  5. 2026LexFlip: A Dissociation Diagnostic for Legal Meaning Preservation Metricscited by 0
  6. 2026Can We Trust the Judges? Validation of Factuality Evaluation Methods via Answer Perturbationcited by 0