# Improving the quality of Web-mined Parallel Corpora of Low-Resource Languages using Debiasing Heuristics

arXiv: https://arxiv.org/abs/2502.19074 · HTML: https://mlsanity.com/paper/2502.19074/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 52 references that are in the map.
- First posted on arXiv 26 Feb 2025.

## Builds on (most cited first, 20 of 20)

- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [A Call for Clarity in Reporting BLEU Scores](https://mlsanity.com/paper/1804.08771/) (2018)
- [fairseq: A Fast, Extensible Toolkit for Sequence Modeling](https://mlsanity.com/paper/1904.01038/) (2019)
- [No Language Left Behind: Scaling Human-Centered Machine Translation](https://mlsanity.com/paper/2207.04672/) (2022)
- [The State and Fate of Linguistic Diversity and Inclusion in the NLP World](https://mlsanity.com/paper/2004.09095/) (2020)
- [Language-agnostic BERT Sentence Embedding](https://mlsanity.com/paper/2007.01852/) (2020)
- [Six Challenges for Neural Machine Translation](https://mlsanity.com/paper/1706.03872/) (2017)
- [Quality at a Glance: An Audit of Web-Crawled Multilingual Datasets](https://mlsanity.com/paper/2103.12028/) (2021)
- [CCMatrix: Mining Billions of High-Quality Parallel Sentences on the WEB](https://mlsanity.com/paper/1911.04944/) (2019)
- [IndicTrans2: Towards High-Quality and Accessible Machine Translation Models for all 22 Scheduled Indian Languages](https://mlsanity.com/paper/2305.16307/) (2023)
- [CCAligned: A Massive Collection of Cross-Lingual Web-Document Pairs](https://mlsanity.com/paper/1911.06154/) (2019)
- [On the Impact of Various Types of Noise on Neural Machine Translation](https://mlsanity.com/paper/1805.12282/) (2018)
- [Some Languages are More Equal than Others: Probing Deeper into the Linguistic Disparity in the NLP World](https://mlsanity.com/paper/2210.08523/) (2022)
- [Survey on Publicly Available Sinhala Natural Language Processing Tools and Research](https://mlsanity.com/paper/1906.02358/) (2019)
- [Low-Resource Corpus Filtering using Multilingual Sentence Embeddings](https://mlsanity.com/paper/1906.08885/) (2019)
- [Data Augmentation and Terminology Integration for Domain-Specific Sinhala-English-Tamil Statistical Machine Translation](https://mlsanity.com/paper/2011.02821/) (2020)
- [Parallel Corpus Filtering via Pre-trained Language Models](https://mlsanity.com/paper/2005.06166/) (2020)
- [Quality Does Matter: A Detailed Look at the Quality and Utility of Web-Mined Parallel Corpora](https://mlsanity.com/paper/2402.07446/) (2024)
- [Analyzing Zero-shot Cross-lingual Transfer in Supervised NLP Tasks](https://mlsanity.com/paper/2101.10649/) (2021)
- [Improving Multilingual Neural Machine Translation System for Indic Languages](https://mlsanity.com/paper/2209.13279/) (2022)

## Built on by (most cited first, 3 of 3)

- [SinLlama -- A Large Language Model for Sinhala](https://mlsanity.com/paper/2508.09115/) (2025)
- [EnSiTa - A Trilingual Multi-Domain Parallel Dataset and Benchmark for Domain-Specific Machine Translation](https://mlsanity.com/paper/2609.29511/) (2026)
- [Trilingual Topic Modeling of Sri Lankan Parliamentary Debates](https://mlsanity.com/paper/2608.20365/) (2026)

## Abstract (arXiv)

Parallel Data Curation (PDC) techniques aim to filter out noisy parallel sentences from web-mined corpora. Ranking sentence pairs using similarity scores on sentence embeddings derived from Pre-trained Multilingual Language Models (multiPLMs) is the most common PDC technique. However, previous research has shown that the choice of the multiPLM significantly impacts the quality of the filtered parallel corpus, and the Neural Machine Translation (NMT) models trained using such data show a disparity across multiPLMs. This paper shows that this disparity is due to different multiPLMs being biased towards certain types of sentence pairs, which are treated as noise from an NMT point of view. We show that such noisy parallel sentences can be removed to a certain extent by employing a series of heuristics. The NMT models, trained using the curated corpus, lead to producing better results while minimizing the disparity across multiPLMs. We publicly release the source code and the curated datasets.