# What Does LLM Refinement Actually Improve? A Systematic Study on Document-Level Literary Translation

arXiv: https://arxiv.org/abs/2605.13368 · HTML: https://mlsanity.com/paper/2605.13368/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 32 references that are in the map.
- First posted on arXiv 13 May 2026.

## Builds on (most cited first, 22 of 22)

- [gpt-oss-120b & gpt-oss-20b Model Card](https://mlsanity.com/paper/2508.10925/) (2025)
- [s1: Simple test-time scaling](https://mlsanity.com/paper/2501.19393/) (2025)
- [SGLang: Efficient Execution of Structured Language Model Programs](https://mlsanity.com/paper/2312.07104/) (2023)
- [Experts, Errors, and Context: A Large-Scale Study of Human Evaluation for Machine Translation](https://mlsanity.com/paper/2104.14478/) (2021)
- [xCOMET: Transparent Machine Translation Evaluation through Fine-grained Error Detection](https://mlsanity.com/paper/2310.10482/) (2023)
- [GEMBA-MQM: Detecting Translation Quality Error Spans with GPT-4](https://mlsanity.com/paper/2310.13988/) (2023)
- [Marco-o1: Towards Open Reasoning Models for Open-Ended Solutions](https://mlsanity.com/paper/2411.14405/) (2024)
- [Sampling-Based Approximations to Minimum Bayes Risk Decoding for Neural Machine Translation](https://mlsanity.com/paper/2108.04718/) (2021)
- [Adapting Large Language Models for Document-Level Machine Translation](https://mlsanity.com/paper/2401.06468/) (2024)
- [LLMRefine: Pinpointing and Refining Large Language Models via Fine-Grained Actionable Feedback](https://mlsanity.com/paper/2311.09336/) (2023)
- [Translating Step-by-Step: Decomposing the Translation Process for Improved Translation Quality of Long-Form Texts](https://mlsanity.com/paper/2409.06790/) (2024)
- [AFRIDOC-MT: Document-level MT Corpus for African Languages](https://mlsanity.com/paper/2501.06374/) (2025)
- [Multilingual Contextualization of Large Language Models for Document-Level Machine Translation](https://mlsanity.com/paper/2504.12140/) (2025)
- [Fine-Grained and Multi-Dimensional Metrics for Document-Level Machine Translation](https://mlsanity.com/paper/2410.20941/) (2024)
- [Investigating Length Issues in Document-level Machine Translation](https://mlsanity.com/paper/2412.17592/) (2024)
- [Remedy: Learning Machine Translation Evaluation from Human Preferences with Reward Modeling](https://mlsanity.com/paper/2504.13630/) (2025)
- [Source-primed Multi-turn Conversation Helps Large Language Models Translate Documents](https://mlsanity.com/paper/2503.10494/) (2025)
- [Please Translate Again: Two Simple Experiments on Whether Human-Like Reasoning Helps Translation](https://mlsanity.com/paper/2506.04521/) (2025)
- [DocHPLT: A Massively Multilingual Document-Level Translation Dataset](https://mlsanity.com/paper/2508.13079/) (2025)
- [Investigating Test-Time Scaling with Reranking for Machine Translation](https://mlsanity.com/paper/2509.19020/) (2025)
- [TASER: Translation Assessment via Systematic Evaluation and Reasoning](https://mlsanity.com/paper/2510.00255/) (2025)
- [Remedy-R: Generative Reasoning for Machine Translation Evaluation without Error Annotations](https://mlsanity.com/paper/2512.18906/) (2025)

## Built on by (most cited first, 2 of 2)

- [Recursive Self-Improvement in AI: From Bounded Self-Refinement to Autonomous Research Loops](https://mlsanity.com/paper/2607.07663/) (2026)
- [Failure-Aware Long-Form Translation: Design and Implementation of a Recoverable LLM Translation System](https://mlsanity.com/paper/2608.09187/) (2026)

## Abstract (arXiv)

Iterative self-refinement is a simple inference-time strategy for machine translation: an LLM revises its own translation over multiple inference-time passes. Yet document-scale refinement remains poorly understood: 1) which pipelines work best, 2) what quality dimensions improve, and 3) how refiners behave. In this paper, we present a systematic study of document-level literary translation, covering nine LLMs and seven language pairs. Across nine translation-refinement granularity combinations and five refinement strategies, we find a robust recipe: document-level MT followed by segment-level refinement yields strong and stable improvements. In contrast, document-level refinement often makes fewer edits and leads to smaller or less reliable gains. Beyond granularity, A simple general refinement prompt consistently outperforms error-specific prompting and evaluate-then-refine schemes. Our large-scale human evaluation shows that refinement gains come primarily from fluency, style, and terminology, with limited and less consistent improvements in adequacy. Experiments varying model strength reveal refinement projects outputs toward the refiner's distribution rather than performing targeted error repair. These findings clarify the mechanisms and limitations of current refinement approaches.