# Findings of the WMT 2023 Shared Task on Discourse-Level Literary Translation: A Fresh Orb in the Cosmos of LLMs

arXiv: https://arxiv.org/abs/2311.03127 · HTML: https://mlsanity.com/paper/2311.03127/

## Key facts

- Cited by 25 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 47 references that are in the map.
- First posted on arXiv 6 Nov 2023.

## Builds on (most cited first, 26 of 26)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [A Call for Clarity in Reporting BLEU Scores](https://mlsanity.com/paper/1804.08771/) (2018)
- [Multilingual Denoising Pre-training for Neural Machine Translation](https://mlsanity.com/paper/2001.08210/) (2020)
- [COMET: A Neural Framework for MT Evaluation](https://mlsanity.com/paper/2009.09025/) (2020)
- [Non-Autoregressive Neural Machine Translation](https://mlsanity.com/paper/1711.02281/) (2017)
- [Six Challenges for Neural Machine Translation](https://mlsanity.com/paper/1706.03872/) (2017)
- [Mega: Moving Average Equipped Gated Attention](https://mlsanity.com/paper/2209.10655/) (2022)
- [Has Machine Translation Achieved Human Parity? A Case for Document-level Evaluation](https://mlsanity.com/paper/1808.07048/) (2018)
- [Understanding and Improving Lexical Choice in Non-Autoregressive Translation](https://mlsanity.com/paper/2012.14583/) (2020)
- [Exploiting Cross-Sentence Context for Neural Machine Translation](https://mlsanity.com/paper/1704.04347/) (2017)
- [Exploring Document-Level Literary Machine Translation with Parallel Paragraphs from World Literature](https://mlsanity.com/paper/2210.14250/) (2022)
- [Progressive Multi-Granularity Training for Non-Autoregressive Translation](https://mlsanity.com/paper/2106.05546/) (2021)
- [Pre-Trained Multilingual Sequence-to-Sequence Models: A Hope for Low-Resource Language Translation?](https://mlsanity.com/paper/2203.08850/) (2022)
- [A Paradigm Shift: The Future of Machine Translation Lies with Large Language Models](https://mlsanity.com/paper/2305.01181/) (2023)
- [TranSmart: A Practical Interactive Machine Translation System](https://mlsanity.com/paper/2105.13072/) (2021)
- [Translating Pro-Drop Languages with Reconstruction Models](https://mlsanity.com/paper/1801.03257/) (2018)
- [One Model to Learn Both: Zero Pronoun Prediction and Translation](https://mlsanity.com/paper/1909.00369/) (2019)
- [What Level of Quality can Neural Machine Translation Attain on Literary Text?](https://mlsanity.com/paper/1801.04962/) (2018)
- [Disco-Bench: A Discourse-Aware Evaluation Benchmark for Language Modelling](https://mlsanity.com/paper/2307.08074/) (2023)
- [A Novel Approach to Dropped Pronoun Translation](https://mlsanity.com/paper/1604.06285/) (2016)
- [A Survey on Zero Pronoun Translation](https://mlsanity.com/paper/2305.10196/) (2023)
- [Learning to Jointly Translate and Predict Dropped Pronouns with a Shared Reconstruction Mechanism](https://mlsanity.com/paper/1810.06195/) (2018)
- [Encoding Sentence Position in Context-Aware Neural Machine Translation with Concatenation](https://mlsanity.com/paper/2302.06459/) (2023)
- [Revisiting Non-Autoregressive Translation at Scale](https://mlsanity.com/paper/2305.16155/) (2023)

## Built on by (most cited first, 25 of 25)

- [(Perhaps) Beyond Human Translation: Harnessing Multi-Agent Collaboration for Translating Ultra-Long Literary Texts](https://mlsanity.com/paper/2405.11804/) (2024)
- [With Greater Text Comes Greater Necessity: Inference-Time Training Helps Long Text Generation](https://mlsanity.com/paper/2401.11504/) (2024)
- [DelTA: An Online Document-Level Translation Agent Based on Multi-Level Memory](https://mlsanity.com/paper/2410.08143/) (2024)
- [How Good Are LLMs for Literary Translation, Really? Literary Translation Evaluation with Humans and LLMs](https://mlsanity.com/paper/2410.18697/) (2024)
- [Salute the Classic: Revisiting Challenges of Machine Translation in the Age of Large Language Models](https://mlsanity.com/paper/2401.08350/) (2024)
- [Findings of the WMT 2024 Shared Task on Discourse-Level Literary Translation](https://mlsanity.com/paper/2412.11732/) (2024)
- [Anchor-based Large Language Models](https://mlsanity.com/paper/2402.07616/) (2024)
- [Multilingual Contextualization of Large Language Models for Document-Level Machine Translation](https://mlsanity.com/paper/2504.12140/) (2025)
- [Creative and Context-Aware Translation of East Asian Idioms with GPT-4](https://mlsanity.com/paper/2410.00988/) (2024)
- [LiTransProQA: an LLM-based Literary Translation evaluation metric with Professional Question Answering](https://mlsanity.com/paper/2505.05423/) (2025)
- [Bridging the Linguistic Divide: A Survey on Leveraging Large Language Models for Machine Translation](https://mlsanity.com/paper/2504.01919/) (2025)
- [M3T: A New Benchmark Dataset for Multi-Modal Document-Level Machine Translation](https://mlsanity.com/paper/2406.08255/) (2024)
- [DITING: A Multi-Agent Evaluation Framework for Benchmarking Web Novel Translation](https://mlsanity.com/paper/2510.09116/) (2025)
- [Steering Large Language Models for Machine Translation Personalization](https://mlsanity.com/paper/2505.16612/) (2025)
- [DiscoX: Benchmarking Discourse-Level Translation task in Expert Domains](https://mlsanity.com/paper/2511.10984/) (2025)
- [A 2-step Framework for Automated Literary Translation Evaluation: Its Promises and Pitfalls](https://mlsanity.com/paper/2412.01340/) (2024)
- [GRAFT: A Graph-based Flow-aware Agentic Framework for Document-level Machine Translation](https://mlsanity.com/paper/2507.03311/) (2025)
- [Building Large-Scale English-Romanian Literary Translation Resources with Open Models](https://mlsanity.com/paper/2509.07829/) (2025)
- [Loong: A Human-Like Long Document Translation Agent with Observe-and-Act Adaptive Context Selection](https://mlsanity.com/paper/2605.30274/) (2026)
- [Automated evaluation of LLMs for effective machine translation of Mandarin Chinese to English](https://mlsanity.com/paper/2603.09998/) (2026)
- [LLMs Should Incorporate Explicit Mechanisms for Human Empathy](https://mlsanity.com/paper/2604.10557/) (2026)
- [Translation with Thought: Difficulty-Adaptive Reasoning via Reinforcement Learning for Multi-Domain Machine Translation](https://mlsanity.com/paper/2607.29287/) (2026)
- [PAMT: Process-Aligned Reinforcement Learning for Multi-Domain Machine Translation](https://mlsanity.com/paper/2608.03077/) (2026)
- [PAUSE: Editable Strategy Artifacts for Long-Form Cultural Story Adaptation](https://mlsanity.com/paper/2608.28633/) (2026)
- [Doc2FRC: Length-Consistent Document-Level Machine Translation via Fixed-Range Chunking](https://mlsanity.com/paper/2609.12674/) (2026)

## Abstract (arXiv)

Translating literary works has perennially stood as an elusive dream in machine translation (MT), a journey steeped in intricate challenges. To foster progress in this domain, we hold a new shared task at WMT 2023, the first edition of the Discourse-Level Literary Translation. First, we (Tencent AI Lab and China Literature Ltd.) release a copyrighted and document-level Chinese-English web novel corpus. Furthermore, we put forth an industry-endorsed criteria to guide human evaluation process. This year, we totally received 14 submissions from 7 academia and industry teams. We employ both automatic and human evaluations to measure the performance of the submitted systems. The official ranking of the systems is based on the overall human judgments. In addition, our extensive analysis reveals a series of interesting findings on literary and discourse-aware MT. We release data, system outputs, and leaderboard at http://www2.statmt.org/wmt23/literary-translation-task.html.