mlsanityData to 14 Oct 2026

Evaluating Communicative Success in Machine-Translated Conversation

arXiv 2609.19885 ↗First posted 17 Sep 2026cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

Interpreter agents built on machine translation (MT) increasingly mediate live conversation between people who do not share a language, yet we still evaluate them with metrics built for isolated sentences, which measure fidelity rather than whether communication succeeds. We introduce a reusable three-layer checklist-and-judge framework that evaluates interpreter-mediated conversation across semantic, pragmatic, and cultural-social dimensions, covering the naturalness, intent, and social appropriateness that fidelity metrics leave unmeasured. It runs in both single-turn and interactive multi-turn settings, where simulated users reply to translated messages as the conversation unfolds and each turn is scored alongside the conversation as a whole. We extensively validate it through controlled perturbations, cross-judge comparisons, and human annotations. Our main single-turn benchmark evaluates 10 interpreter setups across Arabic, Bengali, Indonesian, and Korean from 5,624 OpenSubtitles-derived scenarios spanning 12 translation directions, and our multi-turn study covers all 6 language pairs in scripted and live modes. Results show a consistent decline from semantic to pragmatic and cultural-social success, while conventional MT metrics overlook failures among stronger interpreters, and prompt ablations show that scenario context, structured instructions, and cultural context improve communicative success, although gains vary across setups. Our work thus provides an evaluation framework and benchmark for interpreter agents in conversation, and highlights the importance of communicative success alongside existing translation metrics.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 40 of its 75 references that are in the map. First posted on arXiv 17 Sep 2026.

Builds on

40 of 75 references.

References · 40

  1. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  2. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,512
  3. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,276
  4. 2023FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generationcited by 1,282
  5. 2026DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligencecited by 1,040
  6. 2020COMET: A Neural Framework for MT Evaluationcited by 1,033
  7. 2020Beyond Accuracy: Behavioral Testing of NLP models with CheckListcited by 1,006
  8. 2020The State and Fate of Linguistic Diversity and Inclusion in the NLP Worldcited by 798
  9. 2020Language-agnostic BERT Sentence Embeddingcited by 627
  10. 2021The FLORES-101 Evaluation Benchmark for Low-Resource and Multilingual Machine Translationcited by 560
  11. 2016Google's Multilingual Neural Machine Translation System: Enabling Zero-Shot Translationcited by 469
  12. 2023Large Language Models Are State-of-the-Art Evaluators of Translation Qualitycited by 420
28 more
  1. 2021Experts, Errors, and Context: A Large-Scale Study of Human Evaluation for Machine Translationcited by 338
  2. 2024Replacing Judges with Juries: Evaluating LLM Generations with a Panel of Diverse Modelscited by 326
  3. 2023MEGA: Multilingual Evaluation of Generative AIcited by 293
  4. 2022Challenges and Strategies in Cross-Cultural NLPcited by 218
  5. 2018Dear Sir or Madam, May I introduce the GYAFC Dataset: Corpus, Benchmarks and Metrics for Formality Style Transfercited by 211
  6. 2022CometKiwi: IST-Unbabel 2022 Submission for the Quality Estimation Shared Taskcited by 207
  7. 2020Participatory Research for Low-resourced Machine Translation: A Case Study in African Languagescited by 159
  8. 2024Long-form factuality in large language modelscited by 155
  9. 2023Having Beer after Prayer? Measuring Cultural Bias in Large Language Modelscited by 153
  10. 2023GEMBA-MQM: Detecting Translation Quality Error Spans with GPT-4cited by 146
  11. 2021Neural Machine Translation for Low-Resource Languages: A Surveycited by 109
  12. 2020Politeness Transfer: A Tag and Generate Approachcited by 78
  13. 2020Speech Translation and the End-to-End Promise: Taking Stock of Where We Arecited by 77
  14. 2023Style Over Substance: Evaluation Biases for Large Language Modelscited by 70
  15. 2024EXAONE 3.5: Series of Large Language Models for Real-world Use Casescited by 53
  16. 2023GlotLID: Language Identification for Low-Resource Languagescited by 39
  17. 2023Are Multilingual LLMs Culturally-Diverse Reasoners? An Investigation into Multicultural Proverbs and Sayingscited by 32
  18. 2022Can Transformer be Too Compositional? Analysing Idiom Processing in Neural Machine Translationcited by 28
  19. 2023ChatGPT MT: Competitive for High- (but not Low-) Resource Languagescited by 23
  20. 2024MQM-APE: Toward High-Quality Error Annotation Predictors with Automatic Post-Editing in LLM Translation Evaluatorscited by 22
  21. 2025TigerLLM - A Family of Bangla Large Language Modelscited by 16
  22. 2025Command R7B Arabic: A Small, Enterprise Focused, Multilingual, and Culturally Aware Arabic LLMcited by 16
  23. 2022Extrinsic Evaluation of Machine Translation Metricscited by 14
  24. 2026When Wording Steers the Evaluation: Framing Bias in LLM judgescited by 8
  25. 2026Beyond Literal Mapping: Benchmarking and Improving Non-Literal Translation Evaluationcited by 6
  26. 2026Ask, Don't Judge: Binary Questions for Interpretable LLM Evaluation and Self-Improvementcited by 5
  27. 2025Specification-Aware Machine Translation and Evaluation for Purpose Alignmentcited by 4
  28. 2026Beyond Literal Translation: Evaluating Cultural Effectiveness in Social Media UGCcited by 2

Built on by

Not cited yet.