mlsanityData to 14 Oct 2026

MAS-LitEval : Multi-Agent System for Literary Translation Quality Assessment

arXiv 2506.14199 ↗First posted 17 Jun 2025cs.CLcited by 1

On the map: LLM agents and tool use → General LLM agent frameworks and benchmarks

Abstract (arXiv)

Literary translation requires preserving cultural nuances and stylistic elements, which traditional metrics like BLEU and METEOR fail to assess due to their focus on lexical overlap. This oversight neglects the narrative consistency and stylistic fidelity that are crucial for literary works. To address this, we propose MAS-LitEval, a multi-agent system using Large Language Models (LLMs) to evaluate translations based on terminology, narrative, and style. We tested MAS-LitEval on translations of The Little Prince and A Connecticut Yankee in King Arthur's Court, generated by various LLMs, and compared it to traditional metrics. MAS-LitEval outperformed these metrics, with top models scoring up to 0.890 in capturing literary nuances. This work introduces a scalable, nuanced framework for Translation Quality Assessment (TQA), offering a practical tool for translators and researchers.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in General LLM agent frameworks and benchmarks, one of 16 directions in LLM agents and tool use. Builds on 11 of its 19 references that are in the map. First posted on arXiv 17 Jun 2025.

Builds on

11 of 19 references.

References · 11

  1. 2022No Language Left Behind: Scaling Human-Centered Machine Translationcited by 1,312
  2. 2020COMET: A Neural Framework for MT Evaluationcited by 1,033
  3. 2021Experts, Errors, and Context: A Large-Scale Study of Human Evaluation for Machine Translationcited by 338
  4. 2024Tower: An Open Multilingual Large Language Model for Translation-Related Taskscited by 206
  5. 2023SOLAR 10.7B: Scaling Large Language Models with Simple yet Effective Depth Up-Scalingcited by 172
  6. 2023Scaling up COMETKIWI: Unbabel-IST 2023 Submission for the Quality Estimation Shared Taskcited by 88
  7. 2023Large language models effectively leverage document-level context for literary translation, but critical errors persistcited by 80
  8. 2022Exploring Document-Level Literary Machine Translation with Parallel Paragraphs from World Literaturecited by 45
  9. 2024How Good Are LLMs for Literary Translation, Really? Literary Translation Evaluation with Humans and LLMscited by 23
  10. 2018What Level of Quality can Neural Machine Translation Attain on Literary Text?cited by 11
  11. 2014Formal Consistency Checking over Specifications in Natural Languagescited by 8

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Creativity Bias: How Machine Evaluation Struggles with Creativity in Literary Translationscited by 2

Similar, not cited

Close in wording, no citation either way

  1. 2024(Perhaps) Beyond Human Translation: Harnessing Multi-Agent Collaboration for Translating Ultra-Long Literary Texts
  2. 2025DITING: A Multi-Agent Evaluation Framework for Benchmarking Web Novel Translation
  3. 2026SAMAS: A Spectrum-Guided Multi-Agent System for Achieving Style Fidelity in Literary Translation
  4. 2026Beyond Literal Mapping: Benchmarking and Improving Non-Literal Translation Evaluation
  5. 2024M-MAD: Multidimensional Multi-Agent Debate for Advanced Machine Translation Evaluation
  6. 2025MAATS: A Multi-Agent Automated Translation System Based on MQM Evaluation
  7. 2025HiMATE: A Hierarchical Multi-Agent Framework for Machine Translation Evaluation
  8. 2026Can an Old Dog Be Taught New Tricks? Taking LLMs Beyond Sentence Level Translation