# Beyond Reproduction: A Paired-Task Framework for Assessing LLM Comprehension and Creativity in Literary Translation

arXiv: https://arxiv.org/abs/2604.18169 · HTML: https://mlsanity.com/paper/2604.18169/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 44 references that are in the map.
- First posted on arXiv 20 Apr 2026.

## Builds on (most cited first, 21 of 21)

- [∞Bench: Extending Long Context Evaluation Beyond 100K Tokens](https://mlsanity.com/paper/2402.13718/) (2024)
- [Art or Artifice? Large Language Models and the False Promise of Creativity](https://mlsanity.com/paper/2309.14556/) (2023)
- [One Thousand and One Pairs: A "novel" challenge for long-context language models](https://mlsanity.com/paper/2406.16264/) (2024)
- [LLM Discussion: Enhancing the Creativity of Large Language Models via Discussion Framework and Role-Play](https://mlsanity.com/paper/2405.06373/) (2024)
- [(Perhaps) Beyond Human Translation: Harnessing Multi-Agent Collaboration for Translating Ultra-Long Literary Texts](https://mlsanity.com/paper/2405.11804/) (2024)
- [Exploring the Potential of Large Language Models in Computational Argumentation](https://mlsanity.com/paper/2311.09022/) (2023)
- [NovelQA: Benchmarking Question Answering on Documents Exceeding 200K Tokens](https://mlsanity.com/paper/2403.12766/) (2024)
- [ByGPT5: End-to-End Style-conditioned Poetry Generation with Token-free Language Models](https://mlsanity.com/paper/2212.10474/) (2022)
- [Finding Flawed Fictions: Evaluating Complex Reasoning in Language Models via Plot Hole Detection](https://mlsanity.com/paper/2504.11900/) (2025)
- [How Good Are LLMs for Literary Translation, Really? Literary Translation Evaluation with Humans and LLMs](https://mlsanity.com/paper/2410.18697/) (2024)
- [Creativity in LLM-based Multi-Agent Systems: A Survey](https://mlsanity.com/paper/2505.21116/) (2025)
- [RELIC: Retrieving Evidence for Literary Claims](https://mlsanity.com/paper/2203.10053/) (2022)
- [Small Language Models can Outperform Humans in Short Creative Writing: A Study Comparing SLMs with Humans and LLMs](https://mlsanity.com/paper/2409.11547/) (2024)
- [LLM-based multi-agent poetry generation in non-cooperative environments](https://mlsanity.com/paper/2409.03659/) (2024)
- [The Impact of Post-editing and Machine Translation on Creativity and Reading Experience](https://mlsanity.com/paper/2101.06125/) (2021)
- [LiTransProQA: an LLM-based Literary Translation evaluation metric with Professional Question Answering](https://mlsanity.com/paper/2505.05423/) (2025)
- [Too Long, Didn't Model: Decomposing LLM Long-Context Understanding With Novels](https://mlsanity.com/paper/2505.14925/) (2025)
- [NovelHopQA: Diagnosing Multi-Hop Reasoning Failures in Long Narrative Contexts](https://mlsanity.com/paper/2506.02000/) (2025)
- [Deep Associations, High Creativity: A Simple yet Effective Metric for Evaluating Large Language Models](https://mlsanity.com/paper/2510.12110/) (2025)
- [KRISTEVA: Close Reading as a Novel Task for Benchmarking Interpretive Reasoning](https://mlsanity.com/paper/2505.09825/) (2025)
- [To MT or not to MT: An eye-tracking study on the reception by Dutch readers of different translation and creativity levels](https://mlsanity.com/paper/2504.19850/) (2025)

## Built on by (most cited first, 1 of 1)

- [Beyond Accuracy: Community Perspectives on Machine Translation](https://mlsanity.com/paper/2606.09655/) (2026)

## Abstract (arXiv)

Large language models (LLMs) are increasingly used for creative tasks such as literary translation. Yet translational creativity remains underexplored and is rarely evaluated at scale, while source-text comprehension is typically studied in isolation, despite the fact that, in professional translation, comprehension and creativity are tightly intertwined. We address these gaps with a paired-task framework applied to literary excerpts from 11 books. Task 1 assesses source-text comprehension, and Task 2 evaluates translational creativity through Units of Creative Potential (UCPs), such as metaphors and wordplay. Using a scalable evaluation setup that combines expert human annotations with UCP-based automatic scoring, we benchmark 23 models and four creativity-oriented prompts. Our findings show that strong comprehension does not translate into human-level creativity: models often produce literal or contextually inappropriate renderings, with particularly large gaps for the more distant English-Chinese language pair. Creativity-oriented prompts yield only modest gains, and only one model, Mistral-Large, comes close to human-level creativity (0.167 vs. 0.246). Across all model-prompt combinations, only three exceed a creativity score of 0.1, while the rest remain at or near zero.