mlsanityData to 14 Oct 2026

Global Explainability of BERT-Based Evaluation Metrics by Disentangling along Linguistic Factors

arXiv 2110.04399 ↗First posted 8 Oct 2021cs.CLcited by 25

Not on the map. Why?

Abstract (arXiv)

Evaluation metrics are a key ingredient for progress of text generation systems. In recent years, several BERT-based evaluation metrics have been proposed (including BERTScore, MoverScore, BLEURT, etc.) which correlate much better with human assessment of text generation quality than BLEU or ROUGE, invented two decades ago. However, little is known what these metrics, which are based on black-box language model representations, actually capture (it is typically assumed they model semantic similarity). In this work, we use a simple regression based global explainability technique to disentangle metric scores along linguistic factors, including semantics, syntax, morphology, and lexical overlap. We show that the different metrics capture all aspects to some degree, but that they are all substantially sensitive to lexical overlap, just like BLEU and ROUGE. This exposes limitations of these novelly proposed metrics, which we also highlight in an adversarial test scenario.

Key facts. Cited by 25 papers in the map (data through 14 Oct 2026). Builds on 38 of its 55 references that are in the map. First posted on arXiv 8 Oct 2021.

Builds on

38 of 55 references.

References · 38

  1. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  2. 2017A Unified Approach to Interpreting Model Predictionscited by 6,555
  3. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,512
  4. 2016"Why Should I Trust You?": Explaining the Predictions of Any Classifiercited by 5,070
  5. 2016Enriching Word Vectors with Subword Informationcited by 1,583
  6. 2020BLEURT: Learning Robust Metrics for Text Generationcited by 1,203
  7. 2019BERT Rediscovers the Classical NLP Pipelinecited by 1,059
  8. 2020COMET: A Neural Framework for MT Evaluationcited by 1,033
  9. 2017SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluationcited by 1,001
  10. 2020A Primer in BERTology: What we know about how BERT workscited by 815
  11. 2019Designing and Interpreting Probes with Control Taskscited by 640
  12. 2020Language-agnostic BERT Sentence Embeddingcited by 627
26 more
  1. 2020Making Monolingual Sentence Embeddings Multilingual using Knowledge Distillationcited by 593
  2. 2020BERT-ATTACK: Adversarial Attack Against BERT Using BERTcited by 491
  3. 2018What you can cram into a single vector: Probing sentence embeddings for linguistic propertiescited by 457
  4. 2018Massively Multilingual Sentence Embeddings for Zero-Shot Cross-Lingual Transfer and Beyondcited by 434
  5. 2019MoverScore: Text Generation Evaluating with Contextualized Embeddings and Earth Mover Distancecited by 418
  6. 2019PAWS: Paraphrase Adversaries from Word Scramblingcited by 402
  7. 2020Universal Dependencies v2: An Evergrowing Multilingual Treebank Collectioncited by 271
  8. 2016Fine-grained Analysis of Sentence Embeddings Using Auxiliary Prediction Taskscited by 216
  9. 2020A Survey of the State of Explainable AI for Natural Language Processingcited by 206
  10. 2019Multilingual Universal Sentence Encoder for Semantic Retrievalcited by 193
  11. 2020Tangled up in BLEU: Reevaluating the Evaluation of Automatic Machine Translation Evaluation Metricscited by 158
  12. 2020Probing the Probing Paradigm: Does Probing Accuracy Entail Task Relevance?cited by 151
  13. 2020Multilingual Alignment of Contextual Word Representationscited by 114
  14. 2020BLEU might be Guilty but References are not Innocentcited by 110
  15. 2019Text Processing Like Humans Do: Visually Attacking and Shielding NLP Systemscited by 101
  16. 2021Scientific Credibility of Machine Translation Research: A Meta-Evaluation of 769 Paperscited by 71
  17. 2020SUPERT: Towards New Frontiers in Unsupervised Evaluation Metrics for Multi-Document Summarizationcited by 69
  18. 2020SBERT-WK: A Sentence Embedding Method by Dissecting BERT-based Word Modelscited by 56
  19. 2020MLQE-PE: A Multilingual Quality Estimation and Post-Editing Datasetcited by 47
  20. 2018Learning Cross-Lingual Sentence Representations via a Multi-task Dual-Encoder Modelcited by 46
  21. 2020On the Limitations of Cross-lingual Encoders as Exposed by Reference-Free Machine Translation Evaluationcited by 36
  22. 2021Translation Error Detection as Rationale Extractioncited by 22
  23. 2020TransQuest at WMT2020: Sentence-Level Direct Assessmentcited by 20
  24. 2021BERT-Defense: A Probabilistic Model Based on BERT to Combat Cognitively Inspired Orthographic Adversarial Attackscited by 17
  25. 2020How to Probe Sentence Embeddings in Low-Resource Languages: On Structural Design Choices for Probing Task Evaluationcited by 7
  26. 2020KoBE: Knowledge-Based Machine Translation Evaluationcited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 13; 2023: 4; 2024: 2; 2025: 4; 2026: 2.

25 papers.

Sort:

By area:

  1. 2022Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Textcited by 155
  2. 2022On the Blind Spots of Model-Based Evaluation Metrics for Text Generationcited by 49
  3. 2022Multi-Objective Hyperparameter Optimization in Machine Learning -- An Overviewcited by 37
  4. 2022DiscoScore: Evaluating Text Generation with BERT and Discourse Coherencecited by 36
  5. 2022DEMETR: Diagnosing Evaluation Metrics for Translationcited by 36
  6. 2022SBERT studies Meaning Representations: Decomposing Sentence Embeddings into Explainable Semantic Featurescited by 33
  7. 2025ExPerT: Effective and Explainable Evaluation of Personalized Long-Form Text Generationcited by 26
  8. 2023The Eval4NLP 2023 Shared Task on Prompting Large Language Models as Explainable Metricscited by 23
  9. 2022MENLI: Robust Evaluation Metrics from Natural Language Inferencecited by 18
  10. 2023NLG Evaluation Metrics Beyond Correlation Analysis: An Empirical Metric Preference Checklistcited by 16
  11. 2022USCORE: An Effective Approach to Fully Unsupervised Evaluation Metrics for Machine Translationcited by 15
  12. 2022Towards Explainable Evaluation Metrics for Natural Language Generationcited by 15
13 more
  1. 2023Towards Explainable Evaluation Metrics for Machine Translationcited by 15
  2. 2025Explainability in Practice: A Survey of Explainable NLP Across Various Domainscited by 11
  3. 2022Layer or Representation Space: What makes BERT-based Evaluation Metrics Robust?cited by 10
  4. 2022FastKASSIM: A Fast Tree Kernel-Based Syntactic Similarity Metriccited by 9
  5. 2023Description-Based Text Similaritycited by 7
  6. 2022Towards explainable evaluation of language models on the semantic similarity of visual conceptscited by 5
  7. 2022EffEval: A Comprehensive Evaluation of Efficiency for MT Evaluation Metricscited by 4
  8. 2024PromptOptMe: Error-Aware Prompt Compression for LLM-based MT Evaluation Metricscited by 4
  9. 2024Onco-Retriever: Generative Classifier for Retrieval of EHR Records in Oncologycited by 3
  10. 2025Beyond N-Grams: Rethinking Evaluation Metrics and Strategies for Multilingual Abstractive Summarizationcited by 3
  11. 2026LexChronos: An Agentic Framework for Structured Event Timeline Extraction in Indian Jurisprudencecited by 1
  12. 2025DeMeVa at LeWiDi-2025: Modeling Perspectives with In-Context Learning and Label Distribution Learningcited by 0
  13. 2026Parametric Knowledge in RAG-SFT for Domain-Specific Document Generationcited by 0