mlsanityData to 14 Oct 2026

The Eval4NLP Shared Task on Explainable Quality Estimation: Overview and Results

arXiv 2110.04392 ↗First posted 8 Oct 2021cs.CLcited by 27

Not on the map. Why?

Abstract (arXiv)

In this paper, we introduce the Eval4NLP-2021shared task on explainable quality estimation. Given a source-translation pair, this shared task requires not only to provide a sentence-level score indicating the overall quality of the translation, but also to explain this score by identifying the words that negatively impact translation quality. We present the data, annotation guidelines and evaluation setup of the shared task, describe the six participating systems, and analyze the results. To the best of our knowledge, this is the first shared task on explainable NLP evaluation metrics. Datasets and results are available at https://github.com/eval4nlp/SharedTask2021.

Key facts. Cited by 27 papers in the map (data through 14 Oct 2026). Builds on 30 of its 63 references that are in the map. First posted on arXiv 8 Oct 2021.

Builds on

30 of 63 references.

References · 30

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2017A Unified Approach to Interpreting Model Predictionscited by 6,555
  3. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,512
  4. 2016"Why Should I Trust You?": Explaining the Predictions of Any Classifiercited by 5,070
  5. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,935
  6. 2017Axiomatic Attribution for Deep Networkscited by 3,384
  7. 2013Visualizing and Understanding Convolutional Networkscited by 2,290
  8. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,488
  9. 2017Learning Important Features Through Propagating Activation Differencescited by 1,458
  10. 2017Towards A Rigorous Science of Interpretable Machine Learningcited by 1,416
  11. 2019Attention is not not Explanationcited by 636
  12. 2019ERASER: A Benchmark to Evaluate Rationalized NLP Modelscited by 530
18 more
  1. 2021Experts, Errors, and Context: A Large-Scale Study of Human Evaluation for Machine Translationcited by 338
  2. 2016Not Just a Black Box: Learning Important Features Through Propagating Activation Differencescited by 297
  3. 2020Multilingual Translation with Extensible Multilingual Pretraining and Finetuningcited by 287
  4. 2016Rationalizing Neural Predictionscited by 281
  5. 2020A Survey of the State of Explainable AI for Natural Language Processingcited by 206
  6. 2020A Diagnostic Study of Explainability Techniques for Text Classificationcited by 159
  7. 2020Learning to Faithfully Rationalize by Constructioncited by 113
  8. 2020Restricting the Flow: Information Bottlenecks for Attributioncited by 110
  9. 2019Rethinking Cooperative Rationalization: Introspective Extraction and Complement Controlcited by 96
  10. 2019The relationship between trust in AI and trustworthy machine learning technologiescited by 78
  11. 2020SUPERT: Towards New Frontiers in Unsupervised Evaluation Metrics for Multi-Document Summarizationcited by 69
  12. 2019The FLoRes Evaluation Datasets for Low-Resource Machine Translation: Nepali-English and Sinhala-Englishcited by 62
  13. 2020MLQE-PE: A Multilingual Quality Estimation and Post-Editing Datasetcited by 47
  14. 2020On the Limitations of Cross-lingual Encoders as Exposed by Reference-Free Machine Translation Evaluationcited by 36
  15. 2021Translation Error Detection as Rationale Extractioncited by 22
  16. 2020TransQuest at WMT2020: Sentence-Level Direct Assessmentcited by 20
  17. 2020Detecting Fine-Grained Cross-Lingual Semantic Divergences without Supervision by Learning to Rankcited by 18
  18. 2021Quality Estimation without Human-labeled Datacited by 15

Built on by

Cited by year

Citing papers per year: 2022: 12; 2023: 6; 2024: 3; 2025: 6.

27 papers.

Sort:

By area:

  1. 2022CometKiwi: IST-Unbabel 2022 Submission for the Quality Estimation Shared Taskcited by 207
  2. 2022DiscoScore: Evaluating Text Generation with BERT and Discourse Coherencecited by 36
  3. 2022DEMETR: Diagnosing Evaluation Metrics for Translationcited by 36
  4. 2023AfriMTE and AfriCOMET: Enhancing COMET to Embrace Under-resourced African Languagescited by 24
  5. 2023The Eval4NLP 2023 Shared Task on Prompting Large Language Models as Explainable Metricscited by 23
  6. 2022Learning to Scaffold: Optimizing Model Explanations for Teachingcited by 20
  7. 2024AI-Assisted Human Evaluation of Machine Translationcited by 20
  8. 2022USCORE: An Effective Approach to Fully Unsupervised Evaluation Metrics for Machine Translationcited by 15
  9. 2022Towards Explainable Evaluation Metrics for Natural Language Generationcited by 15
  10. 2023The Inside Story: Towards Better Understanding of Machine Translation Neural Evaluation Metricscited by 15
  11. 2023Towards Explainable Evaluation Metrics for Machine Translationcited by 15
  12. 2022Extrinsic Evaluation of Machine Translation Metricscited by 14
15 more
  1. 2022Report from the NSF Future Directions Workshop on Automatic Evaluation of Dialog: Research Directions and Challengescited by 13
  2. 2023RAMP: Retrieval and Attribute-Marking Enhanced Prompting for Attribute-Controlled Translationcited by 9
  3. 2025AskQE: Question Answering as Automatic Evaluation for Machine Translationcited by 7
  4. 2023Perturbation-based QE: An Explainable, Unsupervised Word-level Quality Estimation Method for Blackbox Machine Translationcited by 6
  5. 2025LaTIM: Measuring Latent Token-to-Token Interactions in Mamba Modelscited by 6
  6. 2025QE4PE: Word-level Quality Estimation for Human Post-Editingcited by 6
  7. 2025An Interdisciplinary Approach to Human-Centered Machine Translationcited by 6
  8. 2022EffEval: A Comprehensive Evaluation of Efficiency for MT Evaluation Metricscited by 4
  9. 2024Word-Level ASR Quality Estimation for Efficient Corpus Sampling and Post-Editing through Analyzing Attentions of a Reference-Free Metriccited by 4
  10. 2025Unsupervised Word-level Quality Estimation for Machine Translation Through the Lens of Annotators (Dis)agreementcited by 4
  11. 2022Rethink about the Word-level Quality Estimation for Machine Translation from Human Judgementcited by 2
  12. 2024Explanation Regularisation through the Lens of Attributionscited by 1
  13. 2025Toward Machine Translation Literacy: How Lay Users Perceive and Rely on Imperfect Translationscited by 1
  14. 2022Reranking Overgenerated Responses for End-to-End Task-Oriented Dialogue Systemscited by 0
  15. 2022BMX: Boosting Natural Language Generation Metrics with Explainabilitycited by 0