A 2-step Framework for Automated Literary Translation Evaluation: Its Promises and Pitfalls
Not on the map. Why?
Abstract (arXiv)
In this work, we propose and evaluate the feasibility of a two-stage pipeline to evaluate literary machine translation, in a fine-grained manner, from English to Korean. The results show that our framework provides fine-grained, interpretable metrics suited for literary translation and obtains a higher correlation with human judgment than traditional machine translation metrics. Nonetheless, it still fails to match inter-human agreement, especially in metrics like Korean Honorifics. We also observe that LLMs tend to favor translations generated by other LLMs, and we highlight the necessity of developing more sophisticated evaluation methods to ensure accurate and culturally sensitive machine translation of literary works.
Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 26 of its 79 references that are in the map. First posted on arXiv 2 Dec 2024.
Builds on
26 of 79 references.
References · 26
- 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
- 2024The Llama 3 Herd of Modelscited by 14,715
- 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
- 2019BERTScore: Evaluating Text Generation with BERTcited by 5,512
- 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,356
- 2021Zero-Shot Text-to-Image Generationcited by 4,288
- 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,276
- 2023FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generationcited by 1,282
- 2020BLEURT: Learning Robust Metrics for Text Generationcited by 1,203
- 2023Large Language Models are not Fair Evaluatorscited by 1,085
- 2020COMET: A Neural Framework for MT Evaluationcited by 1,033
- 2023AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedbackcited by 748
14 more
- 2023ChatEval: Towards Better LLM-based Evaluators through Multi-Agent Debatecited by 742
- 2023Prometheus: Inducing Fine-grained Evaluation Capability in Language Modelscited by 575
- 2023TIFA: Accurate and Interpretable Text-to-Image Faithfulness Evaluation with Question Answeringcited by 428
- 2020Asking and Answering Questions to Evaluate the Factual Consistency of Summariescited by 378
- 2023xCOMET: Transparent Machine Translation Evaluation through Fine-grained Error Detectioncited by 265
- 2024To CoT or not to CoT? Chain-of-thought helps mainly on math and symbolic reasoningcited by 263
- 2023GEMBA-MQM: Detecting Translation Quality Error Spans with GPT-4cited by 146
- 2024Mind Your Step (by Step): Chain-of-Thought can Reduce Performance on Tasks where Thinking Makes Humans Worsecited by 109
- 2023Large language models effectively leverage document-level context for literary translation, but critical errors persistcited by 80
- 2024FABLES: Evaluating faithfulness and content selection in book-length summarizationcited by 65
- 2022Exploring Document-Level Literary Machine Translation with Parallel Paragraphs from World Literaturecited by 45
- 2024MM-Eval: A Multilingual Meta-Evaluation Benchmark for LLM-as-a-Judge and Reward Modelscited by 35
- 2023Findings of the WMT 2023 Shared Task on Discourse-Level Literary Translation: A Fresh Orb in the Cosmos of LLMscited by 25
- 2024How Good Are LLMs for Literary Translation, Really? Literary Translation Evaluation with Humans and LLMscited by 23
Built on by
Citing papers per year: 2025: 1; 2026: 1.
2 papers.