# Fine-Grained and Multi-Dimensional Metrics for Document-Level Machine Translation

arXiv: https://arxiv.org/abs/2410.20941 · HTML: https://mlsanity.com/paper/2410.20941/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 28 references that are in the map.
- First posted on arXiv 28 Oct 2024.

## Builds on (most cited first, 14 of 14)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [Multilingual Denoising Pre-training for Neural Machine Translation](https://mlsanity.com/paper/2001.08210/) (2020)
- [COMET: A Neural Framework for MT Evaluation](https://mlsanity.com/paper/2009.09025/) (2020)
- [How Good Are GPT Models at Machine Translation? A Comprehensive Evaluation](https://mlsanity.com/paper/2302.09210/) (2023)
- [Towards Making the Most of ChatGPT for Machine Translation](https://mlsanity.com/paper/2303.13780/) (2023)
- [BlonDe: An Automatic Evaluation Metric for Document-level Machine Translation](https://mlsanity.com/paper/2103.11878/) (2021)
- [Efficiently Exploring Large Language Models for Document-Level Machine Translation with In-context Learning](https://mlsanity.com/paper/2406.07081/) (2024)
- [Fine-Tuning Large Language Models to Translate: Will a Touch of Noisy Data in Misaligned Languages Suffice?](https://mlsanity.com/paper/2404.14122/) (2024)
- [Enhancing Document-level Translation of Large Language Model via Translation Mixed-instructions](https://mlsanity.com/paper/2401.08088/) (2024)
- [Evaluating Pronominal Anaphora in Machine Translation: An Evaluation Measure and a Test Suite](https://mlsanity.com/paper/1909.00131/) (2019)
- [Importance-Aware Data Augmentation for Document-Level Neural Machine Translation](https://mlsanity.com/paper/2401.15360/) (2024)

## Built on by (most cited first, 10 of 10)

- [AFRIDOC-MT: Document-level MT Corpus for African Languages](https://mlsanity.com/paper/2501.06374/) (2025)
- [Penalizing Length: Uncovering Systematic Bias in Quality Estimation Metrics](https://mlsanity.com/paper/2510.22028/) (2025)
- [Cross-Preference Learning for Sentence-Level and Context-Aware Machine Translation](https://mlsanity.com/paper/2603.25183/) (2026)
- [What Does LLM Refinement Actually Improve? A Systematic Study on Document-Level Literary Translation](https://mlsanity.com/paper/2605.13368/) (2026)
- [Beyond "To whom it may concern": Tailoring Machine Translation to Audience and Intent](https://mlsanity.com/paper/2606.03259/) (2026)
- [Beyond the Sentence: A Survey on Context-Aware Machine Translation with Large Language Models](https://mlsanity.com/paper/2506.07583/) (2025)
- [ForMaT: Dataset for Visually-Grounded Multilingual PDF Translation](https://mlsanity.com/paper/2605.15794/) (2026)
- [AfriScience-MT: Towards Decolonizing Science in Africa through Text Translation](https://mlsanity.com/paper/2605.29741/) (2026)
- [Can an Old Dog Be Taught New Tricks? Taking LLMs Beyond Sentence Level Translation](https://mlsanity.com/paper/2607.14040/) (2026)
- [The Blindness of Document-Level Translation Evaluation](https://mlsanity.com/paper/2609.05949/) (2026)

## Abstract (arXiv)

Large language models (LLMs) have excelled in various NLP tasks, including machine translation (MT), yet most studies focus on sentence-level translation. This work investigates the inherent capability of instruction-tuned LLMs for document-level translation (docMT). Unlike prior approaches that require specialized techniques, we evaluate LLMs by directly prompting them to translate entire documents in a single pass. Our results show that this method improves translation quality compared to translating sentences separately, even without document-level fine-tuning. However, this advantage is not reflected in BLEU scores, which often favor sentence-based translations. We propose using the LLM-as-a-judge paradigm for evaluation, where GPT-4 is used to assess document coherence, accuracy, and fluency in a more nuanced way than n-gram-based metrics. Overall, our work demonstrates that instruction-tuned LLMs can effectively leverage document context for translation. However, we caution against using BLEU scores for evaluating docMT, as they often provide misleading outcomes, failing to capture the quality of document-level translation. Code and the outputs from GPT4-as-a-judge are available at https://github.com/EIT-NLP/BLEUless_DocMT