mlsanityData to 7 Oct 2026

Investigating Length Issues in Document-level Machine Translation

arXiv 2412.17592 ↗First posted 23 Dec 2024cs.CLcited by 10

Not on the map. Why?

Abstract (arXiv)

Transformer architectures are increasingly effective at processing and generating very long chunks of texts, opening new perspectives for document-level machine translation (MT). In this work, we challenge the ability of MT systems to handle texts comprising up to several thousands of tokens. We design and implement a new approach designed to precisely measure the effect of length increments on MT outputs. Our experiments with two representative architectures unambiguously show that (a)~translation performance decreases with the length of the input text; (b)~the position of sentences within the document matters, and translation quality is higher for sentences occurring earlier in a document. We further show that manipulating the distribution of document lengths and of positional embeddings only marginally mitigates such problems. Our results suggest that even though document-level MT is computationally feasible, it does not yet match the performance of sentence-based MT.

Key facts. Cited by 10 papers in the map (data through 7 Oct 2026). Builds on 55 of its 85 references that are in the map. First posted on arXiv 23 Dec 2024.

Builds on

55 of 85 references.

References · 55

  1. 2017Attention Is All You Needcited by 59,932
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  5. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  6. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  7. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,830
  8. 2018A Call for Clarity in Reporting BLEU Scorescited by 1,790
  9. 2018Self-Attention with Relative Position Representationscited by 1,112
  10. 2021Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationcited by 1,085
  11. 2020Multilingual Denoising Pre-training for Neural Machine Translationcited by 1,028
  12. 2020COMET: A Neural Framework for MT Evaluationcited by 1,024
38 more
  1. 2023Extending Context Window of Large Language Models via Positional Interpolationcited by 721
  2. 2015Sequence Level Training with Recurrent Neural Networkscited by 568
  3. 2019Quantifying the Carbon Emissions of Machine Learningcited by 516
  4. 2023How Good Are GPT Models at Machine Translation? A Comprehensive Evaluationcited by 344
  5. 2017Six Challenges for Neural Machine Translationcited by 334
  6. 2023Landmark Attention: Random-Access Infinite Context Length for Transformerscited by 206
  7. 2024Tower: An Open Multilingual Large Language Model for Translation-Related Taskscited by 205
  8. 2020Tangled up in BLEU: Reevaluating the Evaluation of Automatic Machine Translation Evaluation Metricscited by 155
  9. 2020Automatic Machine Translation Evaluation in Many Languages via Zero-Shot Paraphrasingcited by 114
  10. 2023Randomized Positional Encodings Boost Length Generalization of Transformerscited by 114
  11. 2023PoSE: Efficient Context Window Extension of LLMs via Positional Skip-wise Trainingcited by 109
  12. 2018Context-Aware Neural Machine Translation Learns Anaphora Resolutioncited by 97
  13. 2017Neural Machine Translation with Extended Contextcited by 95
  14. 2017Evaluating Discourse Phenomena in Neural Machine Translationcited by 87
  15. 2018Document-Level Neural Machine Translation with Hierarchical Attention Networkscited by 85
  16. 2023Large language models effectively leverage document-level context for literary translation, but critical errors persistcited by 80
  17. 2018Improving the Transformer Translation Model with Document-Level Contextcited by 74
  18. 2019Selective Attention for Context-aware Neural Machine Translationcited by 60
  19. 2019Microsoft Translator at WMT 2019: Towards Large-Scale Document-Level Neural Machine Translationcited by 56
  20. 2021G-Transformer for Document-level Machine Translationcited by 46
  21. 2017Document Context Neural Machine Translation with Memory Networkscited by 41
  22. 2019Context-Aware Monolingual Repair for Neural Machine Translationcited by 41
  23. 2017Learning to Remember Translation History with a Continuous Cachecited by 39
  24. 2021SHAPE: Shifted Absolute Position Embedding for Transformerscited by 37
  25. 2021When Does Translation Require Context? A Data-driven, Multilingual Explorationcited by 36
  26. 2023Length Extrapolation of Transformers: A Survey from the Perspective of Positional Encodingcited by 32
  27. 2020Dynamic Context Selection for Document-level Neural Machine Translation via Reinforcement Learningcited by 25
  28. 2021Do Context-Aware Translation Models Pay the Right Attention?cited by 21
  29. 2023Contextual Refinement of Translations: Large Language Models for Sentence and Document-Level Post-Editingcited by 21
  30. 2022The Curious Case of Absolute Position Embeddingscited by 17
  31. 2023Document Flattening: Beyond Concatenating Context for Document-Level Neural Machine Translationcited by 15
  32. 2020Contextual Neural Machine Translation Improves Translation of Cataphoric Pronounscited by 13
  33. 2021Divide and Rule: Effective Pre-Training for Context-Aware Multi-Encoder Translation Modelscited by 12
  34. 2019Enhancing Context Modeling with a Query-Guided Capsule Network for Document-level Translationcited by 10
  35. 2021DOCmT5: Document-Level Pretraining of Multilingual Language Modelscited by 8
  36. 2023Improving Long Context Document-Level Machine Translationcited by 8
  37. 2019Context in Neural Machine Translation: A Review of Models and Evaluationscited by 7
  38. 2021A Comparison of Approaches to Document-level Machine Translationcited by 7

50 most cited of 55 All 55 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 3; 2026: 7.

10 papers.

  1. 2025DocHPLT: A Massively Multilingual Document-Level Translation Datasetcited by 7
  2. 2025Optimising ChatGPT for creativity in literary translation: A case study from English into Dutch, Chinese, Catalan and Spanishcited by 2
  3. 2025XToM: Exploring the Multilingual Theory of Mind for Large Language Modelscited by 2
  4. 2026What Does LLM Refinement Actually Improve? A Systematic Study on Document-Level Literary Translationcited by 2
  5. 2026STAR : Sentence Translation Alignment Rate for Document-to-Document Machine Translationcited by 1
  6. 2026Hindsight Quality Prediction Experiments in Multi-Candidate Human-Post-Edited Machine Translationcited by 0
  7. 2026Failure-Aware Long-Form Translation: Design and Implementation of a Recoverable LLM Translation Systemcited by 0
  8. 2026Doc2FRC: Length-Consistent Document-Level Machine Translation via Fixed-Range Chunkingcited by 0
  9. 2026When Do Agents Help? Embedding, LLM and Agentic Alignment of Classical Texts and Their Translationscited by 0
  10. 2026TermJudge: A Document-Level Metric Judging, Not Counting, Terminology in Machine Translation Evaluationcited by 0