# Recovering document annotations for sentence-level bitext

arXiv: https://arxiv.org/abs/2406.03869 · HTML: https://mlsanity.com/paper/2406.03869/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 50 references that are in the map.
- First posted on arXiv 6 Jun 2024.

## Builds on (most cited first, 27 of 27)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Longformer: The Long-Document Transformer](https://mlsanity.com/paper/2004.05150/) (2020)
- [SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing](https://mlsanity.com/paper/1808.06226/) (2018)
- [A Call for Clarity in Reporting BLEU Scores](https://mlsanity.com/paper/1804.08771/) (2018)
- [COMET: A Neural Framework for MT Evaluation](https://mlsanity.com/paper/2009.09025/) (2020)
- [Beyond English-Centric Multilingual Machine Translation](https://mlsanity.com/paper/2010.11125/) (2020)
- [The FLORES-101 Evaluation Benchmark for Low-Resource and Multilingual Machine Translation](https://mlsanity.com/paper/2106.03193/) (2021)
- [WikiMatrix: Mining 135M Parallel Sentences in 1620 Language Pairs from Wikipedia](https://mlsanity.com/paper/1907.05791/) (2019)
- [CometKiwi: IST-Unbabel 2022 Submission for the Quality Estimation Shared Task](https://mlsanity.com/paper/2209.06243/) (2022)
- [Marian: Fast Neural Machine Translation in C++](https://mlsanity.com/paper/1804.00344/) (2018)
- [CCMatrix: Mining Billions of High-Quality Parallel Sentences on the WEB](https://mlsanity.com/paper/1911.04944/) (2019)
- [CCAligned: A Massive Collection of Cross-Lingual Web-Document Pairs](https://mlsanity.com/paper/1911.06154/) (2019)
- [Context-Aware Neural Machine Translation Learns Anaphora Resolution](https://mlsanity.com/paper/1805.10163/) (2018)
- [Neural Machine Translation with Extended Context](https://mlsanity.com/paper/1708.05943/) (2017)
- [Evaluating Discourse Phenomena in Neural Machine Translation](https://mlsanity.com/paper/1711.00513/) (2017)
- [Document-Level Neural Machine Translation with Hierarchical Attention Networks](https://mlsanity.com/paper/1809.01576/) (2018)
- [The FLoRes Evaluation Datasets for Low-Resource Machine Translation: Nepali-English and Sinhala-English](https://mlsanity.com/paper/1902.01382/) (2019)
- [BlonDe: An Automatic Evaluation Metric for Document-level Machine Translation](https://mlsanity.com/paper/2103.11878/) (2021)
- [Embarrassingly Easy Document-Level MT Metrics: How to Convert Any Pretrained Metric Into a Document-Level Metric](https://mlsanity.com/paper/2209.13654/) (2022)
- [Learning Joint Multilingual Sentence Representations with Neural Machine Translation](https://mlsanity.com/paper/1704.04154/) (2017)
- [A Shocking Amount of the Web is Machine Translated: Insights from Multi-Way Parallelism](https://mlsanity.com/paper/2401.05749/) (2024)
- [Escaping the sentence-level paradigm in machine translation](https://mlsanity.com/paper/2304.12959/) (2023)
- [There's no Data Like Better Data: Using QE Metrics for MT Data Filtering](https://mlsanity.com/paper/2311.05350/) (2023)
- [XLEnt: Mining a Large Cross-lingual Entity Dataset with Lexical-Semantic-Phonetic Word Alignment](https://mlsanity.com/paper/2104.08597/) (2021)
- [Identifying Context-Dependent Translations for Evaluation Set Production](https://mlsanity.com/paper/2311.02321/) (2023)
- [Document-Level Language Models for Machine Translation](https://mlsanity.com/paper/2310.12303/) (2023)
- [SLIDE: Reference-free Evaluation for Machine Translation using a Sliding Document Window](https://mlsanity.com/paper/2309.08832/) (2023)

## Built on by (most cited first, 6 of 6)

- [EuroLLM: Multilingual Language Models for Europe](https://mlsanity.com/paper/2409.16235/) (2024)
- [Tower+: Bridging Generality and Translation Specialization in Multilingual LLMs](https://mlsanity.com/paper/2506.17080/) (2025)
- [EuroLLM-22B: Technical Report](https://mlsanity.com/paper/2602.05879/) (2026)
- [DocHPLT: A Massively Multilingual Document-Level Translation Dataset](https://mlsanity.com/paper/2508.13079/) (2025)
- [MLLP-VRAIN UPV system for the IWSLT 2025 Simultaneous Speech Translation Translation task](https://mlsanity.com/paper/2506.18828/) (2025)
- [Doc2FRC: Length-Consistent Document-Level Machine Translation via Fixed-Range Chunking](https://mlsanity.com/paper/2609.12674/) (2026)

## Abstract (arXiv)

Data availability limits the scope of any given task. In machine translation, historical models were incapable of handling longer contexts, so the lack of document-level datasets was less noticeable. Now, despite the emergence of long-sequence methods, we remain within a sentence-level paradigm and without data to adequately approach context-aware machine translation. Most large-scale datasets have been processed through a pipeline that discards document-level metadata. In this work, we reconstruct document-level information for three (ParaCrawl, News Commentary, and Europarl) large datasets in German, French, Spanish, Italian, Polish, and Portuguese (paired with English). We then introduce a document-level filtering technique as an alternative to traditional bitext filtering. We present this filtering with analysis to show that this method prefers context-consistent translations rather than those that may have been sentence-level machine translated. Last we train models on these longer contexts and demonstrate improvement in document-level translation without degradation of sentence-level translation. We release our dataset, ParaDocs, and resulting models as a resource to the community.