# BMX: Boosting Natural Language Generation Metrics with Explainability

arXiv: https://arxiv.org/abs/2212.10469 · HTML: https://mlsanity.com/paper/2212.10469/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 23 of its 50 references that are in the map.
- First posted on arXiv 20 Dec 2022.

## Builds on (most cited first, 23 of 23)

- [A Unified Approach to Interpreting Model Predictions](https://mlsanity.com/paper/1705.07874/) (2017)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [Explainable Artificial Intelligence (XAI): Concepts, Taxonomies, Opportunities and Challenges toward Responsible AI](https://mlsanity.com/paper/1910.10045/) (2019)
- [XNLI: Evaluating Cross-lingual Sentence Representations](https://mlsanity.com/paper/1809.05053/) (2018)
- [BARTScore: Evaluating Generated Text as Text Generation](https://mlsanity.com/paper/2106.11520/) (2021)
- [SummEval: Re-evaluating Summarization Evaluation](https://mlsanity.com/paper/2007.12626/) (2020)
- [Making Monolingual Sentence Embeddings Multilingual using Knowledge Distillation](https://mlsanity.com/paper/2004.09813/) (2020)
- [BERT-ATTACK: Adversarial Attack Against BERT Using BERT](https://mlsanity.com/paper/2004.09984/) (2020)
- [Experts, Errors, and Context: A Large-Scale Study of Human Evaluation for Machine Translation](https://mlsanity.com/paper/2104.14478/) (2021)
- [Understanding Neural Networks through Representation Erasure](https://mlsanity.com/paper/1612.08220/) (2016)
- [CometKiwi: IST-Unbabel 2022 Submission for the Quality Estimation Shared Task](https://mlsanity.com/paper/2209.06243/) (2022)
- [Evaluating Explainable AI: Which Algorithmic Explanations Help Users Predict Model Behavior?](https://mlsanity.com/paper/2005.01831/) (2020)
- [Re-evaluating Evaluation in Text Summarization](https://mlsanity.com/paper/2010.07100/) (2020)
- [A Statistical Analysis of Summarization Evaluation Metrics using Resampling Methods](https://mlsanity.com/paper/2104.00054/) (2021)
- [TransQuest: Translation Quality Estimation with Cross-lingual Transformers](https://mlsanity.com/paper/2011.01536/) (2020)
- [Perturbation CheckLists for Evaluating NLG Evaluation Metrics](https://mlsanity.com/paper/2109.05771/) (2021)
- [MLQE-PE: A Multilingual Quality Estimation and Post-Editing Dataset](https://mlsanity.com/paper/2010.04480/) (2020)
- [Interpretation of NLP models through input marginalization](https://mlsanity.com/paper/2010.13984/) (2020)
- [Replicability Analysis for Natural Language Processing: Testing Significance with Multiple Datasets](https://mlsanity.com/paper/1709.09500/) (2017)
- [The Eval4NLP Shared Task on Explainable Quality Estimation: Overview and Results](https://mlsanity.com/paper/2110.04392/) (2021)
- [An Exploratory Analysis of Multilingual Word-Level Quality Estimation with Cross-Lingual Transformers](https://mlsanity.com/paper/2106.00143/) (2021)
- [USCORE: An Effective Approach to Fully Unsupervised Evaluation Metrics for Machine Translation](https://mlsanity.com/paper/2202.10062/) (2022)

## Abstract (arXiv)

State-of-the-art natural language generation evaluation metrics are based on black-box language models. Hence, recent works consider their explainability with the goals of better understandability for humans and better metric analysis, including failure cases. In contrast, our proposed method BMX: Boosting Natural Language Generation Metrics with explainability explicitly leverages explanations to boost the metrics' performance. In particular, we perceive feature importance explanations as word-level scores, which we convert, via power means, into a segment-level score. We then combine this segment-level score with the original metric to obtain a better metric. Our tests show improvements for multiple metrics across MT and summarization datasets. While improvements in machine translation are small, they are strong for summarization. Notably, BMX with the LIME explainer and preselected parameters achieves an average improvement of 0.087 points in Spearman correlation on the system-level evaluation of SummEval.