# Separating the Wheat from the Chaff with BREAD: An open-source benchmark and metrics to detect redundancy in text

arXiv: https://arxiv.org/abs/2311.06440 · HTML: https://mlsanity.com/paper/2311.06440/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 57 references that are in the map.
- First posted on arXiv 11 Nov 2023.

## Builds on (most cited first, 27 of 27)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [PaLM: Scaling Language Modeling with Pathways](https://mlsanity.com/paper/2204.02311/) (2022)
- [A Call for Clarity in Reporting BLEU Scores](https://mlsanity.com/paper/1804.08771/) (2018)
- [A Closer Look at Memorization in Deep Networks](https://mlsanity.com/paper/1706.05394/) (2017)
- [What Neural Networks Memorize and Why: Discovering the Long Tail via Influence Estimation](https://mlsanity.com/paper/2008.03703/) (2020)
- [Texygen: A Benchmarking Platform for Text Generation Models](https://mlsanity.com/paper/1802.01886/) (2018)
- [Quality at a Glance: An Audit of Web-Crawled Multilingual Datasets](https://mlsanity.com/paper/2103.12028/) (2021)
- [Task2Vec: Task Embedding for Meta-Learning](https://mlsanity.com/paper/1902.03545/) (2019)
- [MADLAD-400: A Multilingual And Document-Level Large Audited Dataset](https://mlsanity.com/paper/2309.04662/) (2023)
- [To Ship or Not to Ship: An Extensive Evaluation of Automatic Metrics for Machine Translation](https://mlsanity.com/paper/2107.10821/) (2021)
- [Language ID in the Wild: Unexpected Challenges on the Path to a Thousand-Language Web Text Corpus](https://mlsanity.com/paper/2010.14571/) (2020)
- [On the Impact of Various Types of Noise on Neural Machine Translation](https://mlsanity.com/paper/1805.12282/) (2018)
- [Building Machine Translation Systems for the Next Thousand Languages](https://mlsanity.com/paper/2205.03983/) (2022)
- [Epsilon Sampling Rocks: Investigating Sampling Strategies for Minimum Bayes Risk Decoding for Machine Translation](https://mlsanity.com/paper/2305.09860/) (2023)
- [Denoising Neural Machine Translation Training with Trusted Data and Online Data Selection](https://mlsanity.com/paper/1809.00068/) (2018)
- [Dual Conditional Cross-Entropy Filtering of Noisy Parallel Corpora](https://mlsanity.com/paper/1809.00197/) (2018)
- [Dynamic Data Selection for Neural Machine Translation](https://mlsanity.com/paper/1708.00712/) (2017)
- [Low-Resource Corpus Filtering using Multilingual Sentence Embeddings](https://mlsanity.com/paper/1906.08885/) (2019)
- [Dynamically Composing Domain-Data Selection with Clean-Data Selection by "Co-Curricular Learning" for Neural Machine Translation](https://mlsanity.com/paper/1906.01130/) (2019)
- [A Survey of Data Quality Measurement and Monitoring Tools](https://mlsanity.com/paper/1907.08138/) (2019)
- [Cynical Selection of Language Model Training Data](https://mlsanity.com/paper/1709.02279/) (2017)
- [Extending F1 metric, probabilistic approach](https://mlsanity.com/paper/2210.11997/) (2022)
- [The Curse of Low Task Diversity: On the Failure of Transfer Learning to Outperform MAML and Their Empirical Equivalence](https://mlsanity.com/paper/2208.01545/) (2022)
- [Dynamic Sentence Sampling for Efficient Training of Neural Machine Translation](https://mlsanity.com/paper/1805.00178/) (2018)
- [Microsoft's Submission to the WMT2018 News Translation Task: How I Learned to Stop Worrying and Love the Data](https://mlsanity.com/paper/1809.00196/) (2018)
- [Boosting Neural Machine Translation](https://mlsanity.com/paper/1612.06138/) (2016)

## Built on by (most cited first, 1 of 1)

- [SMOL: Professionally translated parallel data for 115 under-represented languages](https://mlsanity.com/paper/2502.12301/) (2025)

## Abstract (arXiv)

Data quality is a problem that perpetually resurfaces throughout the field of NLP, regardless of task, domain, or architecture, and remains especially severe for lower-resource languages. A typical and insidious issue, affecting both training data and model output, is data that is repetitive and dominated by linguistically uninteresting boilerplate, such as price catalogs or computer-generated log files. Though this problem permeates many web-scraped corpora, there has yet to be a benchmark to test against, or a systematic study to find simple metrics that generalize across languages and agree with human judgements of data quality. In the present work, we create and release BREAD, a human-labeled benchmark on repetitive boilerplate vs. plausible linguistic content, spanning 360 languages. We release several baseline CRED (Character REDundancy) scores along with it, and evaluate their effectiveness on BREAD. We hope that the community will use this resource to develop better filtering methods, and that our reference implementations of CRED scores can become standard corpus evaluation tools, driving the development of cleaner language modeling corpora, especially in low-resource languages.