mlsanityData to 14 Oct 2026

AskQE: Question Answering as Automatic Evaluation for Machine Translation

arXiv 2504.11582 ↗First posted 15 Apr 2025cs.CLcited by 7

Not on the map. Why?

Abstract (arXiv)

How can a monolingual English speaker determine whether an automatic translation in French is good enough to be shared? Existing MT error detection and quality estimation (QE) techniques do not address this practical scenario. We introduce AskQE, a question generation and answering framework designed to detect critical MT errors and provide actionable feedback, helping users decide whether to accept or reject MT outputs even without the knowledge of the target language. Using ContraTICO, a dataset of contrastive synthetic MT errors in the COVID-19 domain, we explore design choices for AskQE and develop an optimized version relying on LLaMA-3 70B and entailed facts to guide question generation. We evaluate the resulting system on the BioMQM dataset of naturally occurring MT errors, where AskQE has higher Kendall's Tau correlation and decision accuracy with human ratings compared to other QE metrics.

Key facts. Cited by 7 papers in the map (data through 14 Oct 2026). Builds on 32 of its 70 references that are in the map. First posted on arXiv 15 Apr 2025.

Builds on

32 of 70 references.

References · 32

  1. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  2. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,512
  3. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,148
  4. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,998
  5. 2017Explanation in Artificial Intelligence: Insights from the Social Sciencescited by 1,206
  6. 2024Yi: Open Foundation Models by 01.AIcited by 719
  7. 2019BLiMP: The Benchmark of Linguistic Minimal Pairs for Englishcited by 564
  8. 2020Asking and Answering Questions to Evaluate the Factual Consistency of Summariescited by 378
  9. 2021Experts, Errors, and Context: A Large-Scale Study of Human Evaluation for Machine Translationcited by 338
  10. 2020FEQA: A Question Answering Evaluation Framework for Faithfulness Assessment in Abstractive Summarizationcited by 293
  11. 2023xCOMET: Transparent Machine Translation Evaluation through Fine-grained Error Detectioncited by 265
  12. 2022Towards a Unified Multi-Dimensional Evaluator for Text Generationcited by 264
20 more
  1. 2021QAFactEval: Improved QA-Based Factual Consistency Evaluation for Summarizationcited by 236
  2. 2023GEMBA-MQM: Detecting Translation Quality Error Spans with GPT-4cited by 146
  3. 2023The Devil is in the Errors: Leveraging Large Language Models for Fine-grained Machine Translation Evaluationcited by 95
  4. 2024MetricX-24: The Google Submission to the WMT 2024 Metrics Shared Taskcited by 93
  5. 2020Towards Question-Answering as an Automatic Metric for Evaluating the Content Quality of a Summarycited by 88
  6. 2023TIGERScore: Towards Building Explainable Metric for All Text Generation Taskscited by 70
  7. 2020TICO-19: the Translation Initiative for Covid-19cited by 68
  8. 2021Perturbation CheckLists for Evaluating NLG Evaluation Metricscited by 50
  9. 2020Synthetic Data Augmentation for Zero-Shot Cross-Lingual Question Answeringcited by 41
  10. 2021Controllable Open-ended Question Generation with A New Question Type Ontologycited by 36
  11. 2022DEMETR: Diagnosing Evaluation Metrics for Translationcited by 36
  12. 2021The Eval4NLP Shared Task on Explainable Quality Estimation: Overview and Resultscited by 27
  13. 2021Translation Quality Assessment: A Brief Survey on Manual and Automatic Methodscited by 23
  14. 2021Translation Error Detection as Rationale Extractioncited by 22
  15. 2023Physician Detection of Clinical Harm in Machine Translation: Quality Estimation Aids in Reliance and Backtranslation Identifies Critical Errorscited by 22
  16. 2023Atomic Inference for NLI with Generated Facts as Atomscited by 18
  17. 2020Towards Zero-Shot Multilingual Synthetic Question and Answer Generation for Cross-Lingual Reading Comprehensioncited by 17
  18. 2024QGEval: Benchmarking Multi-dimensional Evaluation for Question Generationcited by 17
  19. 2024InfoLossQA: Characterizing and Recovering Information Loss in Text Simplificationcited by 14
  20. 2025Do LLMs Understand Your Translations? Evaluating Paragraph-level MT with Question Answeringcited by 10

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 5; 2026: 2.

7 papers.

  1. 2025LiTransProQA: an LLM-based Literary Translation evaluation metric with Professional Question Answeringcited by 10
  2. 2025An Interdisciplinary Approach to Human-Centered Machine Translationcited by 6
  3. 2025Should I Share this Translation? Evaluating Quality Feedback for User Reliance on Machine Translationcited by 5
  4. 2025Discrepancy Detection at the Data Level: Toward Consistent Multilingual Question Answeringcited by 2
  5. 2025Toward Machine Translation Literacy: How Lay Users Perceive and Rely on Imperfect Translationscited by 1
  6. 2026Ouvia: A User-centered Framework for Measuring Usability of Speech Translation in Real-World Communication Scenarioscited by 0
  7. 2026Beyond BLEU: A Case for Redefining Sign Language Translation Benchmarkscited by 0