# An Unsupervised Method for Building Sentence Simplification Corpora in Multiple Languages

arXiv: https://arxiv.org/abs/2109.00165 · HTML: https://mlsanity.com/paper/2109.00165/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 54 references that are in the map.
- First posted on arXiv 1 Sep 2021.

## Builds on (most cited first, 25 of 25)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [A Call for Clarity in Reporting BLEU Scores](https://mlsanity.com/paper/1804.08771/) (2018)
- [fairseq: A Fast, Extensible Toolkit for Sequence Modeling](https://mlsanity.com/paper/1904.01038/) (2019)
- [Multilingual Denoising Pre-training for Neural Machine Translation](https://mlsanity.com/paper/2001.08210/) (2020)
- [Improving Neural Machine Translation Models with Monolingual Data](https://mlsanity.com/paper/1511.06709/) (2015)
- [CCNet: Extracting High Quality Monolingual Datasets from Web Crawl Data](https://mlsanity.com/paper/1911.00359/) (2019)
- [Scaling Neural Machine Translation](https://mlsanity.com/paper/1806.00187/) (2018)
- [Facebook FAIR's WMT19 News Translation Task Submission](https://mlsanity.com/paper/1907.06616/) (2019)
- [Sentence Simplification with Deep Reinforcement Learning](https://mlsanity.com/paper/1703.10931/) (2017)
- [ASSET: A Dataset for Tuning and Evaluation of Sentence Simplification Models with Multiple Rewriting Transformations](https://mlsanity.com/paper/2005.00481/) (2020)
- [BLEU is Not Suitable for the Evaluation of Text Simplification](https://mlsanity.com/paper/1810.05995/) (2018)
- [Controllable Sentence Simplification](https://mlsanity.com/paper/1910.02677/) (2019)
- [EditNTS: An Neural Programmer-Interpreter Model for Sentence Simplification through Explicit Editing](https://mlsanity.com/paper/1906.08104/) (2019)
- [Unsupervised Statistical Machine Translation](https://mlsanity.com/paper/1809.01272/) (2018)
- [ParaNMT-50M: Pushing the Limits of Paraphrastic Sentence Embeddings with Millions of Machine Translations](https://mlsanity.com/paper/1711.05732/) (2017)
- [ParaBank: Monolingual Bitext Generation and Sentential Paraphrasing via Lexically-constrained Neural Machine Translation](https://mlsanity.com/paper/1901.03644/) (2019)
- [Short Text Topic Modeling Techniques, Applications, and Performance: A Survey](https://mlsanity.com/paper/1904.07695/) (2019)
- [Iterative Edit-Based Unsupervised Sentence Simplification](https://mlsanity.com/paper/2006.09639/) (2020)
- [Unsupervised Neural Text Simplification](https://mlsanity.com/paper/1810.07931/) (2018)
- [Integrating Transformer and Paraphrase Rules for Sentence Simplification](https://mlsanity.com/paper/1810.11193/) (2018)
- [Improving Neural Response Diversity with Frequency-Aware Cross-Entropy Loss](https://mlsanity.com/paper/1902.09191/) (2019)
- [Learning Paraphrastic Sentence Embeddings from Back-Translated Bitext](https://mlsanity.com/paper/1706.01847/) (2017)
- [Token-level Adaptive Training for Neural Machine Translation](https://mlsanity.com/paper/2010.04380/) (2020)
- [MUSS: Multilingual Unsupervised Sentence Simplification by Mining Paraphrases](https://mlsanity.com/paper/2005.00352/) (2020)
- [Unsupervised Sentence Simplification Using Deep Semantics](https://mlsanity.com/paper/1507.08452/) (2015)

## Built on by (most cited first, 14 of 14)

- [Sentence Simplification via Large Language Models](https://mlsanity.com/paper/2302.11957/) (2023)
- [Preparing the Vuk'uzenzele and ZA-gov-multilingual South African multilingual corpora](https://mlsanity.com/paper/2303.03750/) (2023)
- [Learning to Paraphrase Sentences to Different Complexity Levels](https://mlsanity.com/paper/2308.02226/) (2023)
- [Chinese Idiom Paraphrasing](https://mlsanity.com/paper/2204.07555/) (2022)
- [Exploiting Summarization Data to Help Text Simplification](https://mlsanity.com/paper/2302.07124/) (2023)
- [A New Dataset and Empirical Study for Sentence Simplification in Chinese](https://mlsanity.com/paper/2306.04188/) (2023)
- [Creating a silver standard for patent simplification](https://mlsanity.com/paper/2310.15689/) (2023)
- [Label Confidence Weighted Learning for Target-level Sentence Simplification](https://mlsanity.com/paper/2410.05748/) (2024)
- [Readability-guided Idiom-aware Sentence Simplification (RISS) for Chinese](https://mlsanity.com/paper/2406.02974/) (2024)
- [Sentence Simplification Using Paraphrase Corpus for Initialization](https://mlsanity.com/paper/2305.19754/) (2023)
- [MCTS: A Multi-Reference Chinese Text Simplification Dataset](https://mlsanity.com/paper/2306.02796/) (2023)
- [System Report for CCL24-Eval Task 7: Multi-Error Modeling and Fluency-Targeted Pre-training for Chinese Essay Evaluation](https://mlsanity.com/paper/2407.08206/) (2024)
- [OasisSimp: An Open-source Asian-English Sentence Simplification Dataset](https://mlsanity.com/paper/2603.14111/) (2026)
- [ComplexityMT: Benchmarking the Interaction Between Text Complexity and Machine Translation](https://mlsanity.com/paper/2606.05421/) (2026)

## Abstract (arXiv)

The availability of parallel sentence simplification (SS) is scarce for neural SS modelings. We propose an unsupervised method to build SS corpora from large-scale bilingual translation corpora, alleviating the need for SS supervised corpora. Our method is motivated by the following two findings: neural machine translation model usually tends to generate more high-frequency tokens and the difference of text complexity levels exists between the source and target language of a translation corpus. By taking the pair of the source sentences of translation corpus and the translations of their references in a bridge language, we can construct large-scale pseudo parallel SS data. Then, we keep these sentence pairs with a higher complexity difference as SS sentence pairs. The building SS corpora with an unsupervised approach can satisfy the expectations that the aligned sentences preserve the same meanings and have difference in text complexity levels. Experimental results show that SS methods trained by our corpora achieve the state-of-the-art results and significantly outperform the results on English benchmark WikiLarge.