# Teaching the Pre-trained Model to Generate Simple Texts for Text Simplification

arXiv: https://arxiv.org/abs/2305.12463 · HTML: https://mlsanity.com/paper/2305.12463/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 50 references that are in the map.
- First posted on arXiv 21 May 2023.

## Builds on (most cited first, 25 of 25)

- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [Scaling Instruction-Finetuned Language Models](https://mlsanity.com/paper/2210.11416/) (2022)
- [Don't Stop Pretraining: Adapt Language Models to Domains and Tasks](https://mlsanity.com/paper/2004.10964/) (2020)
- [fairseq: A Fast, Extensible Toolkit for Sequence Modeling](https://mlsanity.com/paper/1904.01038/) (2019)
- [Text Summarization with Pretrained Encoders](https://mlsanity.com/paper/1908.08345/) (2019)
- [Sentence Simplification with Deep Reinforcement Learning](https://mlsanity.com/paper/1703.10931/) (2017)
- [Neural CRF Model for Sentence Alignment in Text Simplification](https://mlsanity.com/paper/2005.02324/) (2020)
- [BLEU is Not Suitable for the Evaluation of Text Simplification](https://mlsanity.com/paper/1810.05995/) (2018)
- [EditNTS: An Neural Programmer-Interpreter Model for Sentence Simplification through Explicit Editing](https://mlsanity.com/paper/1906.08104/) (2019)
- [Controllable Text Simplification with Explicit Paraphrasing](https://mlsanity.com/paper/2010.11004/) (2020)
- [Keep it Simple: Unsupervised Simplification of Multi-Paragraph Text](https://mlsanity.com/paper/2107.03444/) (2021)
- [Dict-BERT: Enhancing Language Model Pre-training with Dictionary](https://mlsanity.com/paper/2110.06490/) (2021)
- [Train No Evil: Selective Masking for Task-Guided Pre-Training](https://mlsanity.com/paper/2004.09733/) (2020)
- [Evaluating Factuality in Text Simplification](https://mlsanity.com/paper/2204.07562/) (2022)
- [Text Simplification by Tagging](https://mlsanity.com/paper/2103.05070/) (2021)
- [Document-Level Text Simplification: Dataset, Criteria and Baseline](https://mlsanity.com/paper/2110.05071/) (2021)
- [A Word-Complexity Lexicon and A Neural Readability Ranking Model for Lexical Simplification](https://mlsanity.com/paper/1810.05754/) (2018)
- [CompLex: A New Corpus for Lexical Complexity Prediction from Likert Scale Data](https://mlsanity.com/paper/2003.07008/) (2020)
- [SemEval-2021 Task 1: Lexical Complexity Prediction](https://mlsanity.com/paper/2106.00473/) (2021)
- [DEEP: DEnoising Entity Pre-training for Neural Machine Translation](https://mlsanity.com/paper/2111.07393/) (2021)
- [LSBert: A Simple Framework for Lexical Simplification](https://mlsanity.com/paper/2006.14939/) (2020)
- [Exploiting Summarization Data to Help Text Simplification](https://mlsanity.com/paper/2302.07124/) (2023)

## Built on by (most cited first, 10 of 10)

- [BLESS: Benchmarking Large Language Models on Sentence Simplification](https://mlsanity.com/paper/2310.15773/) (2023)
- [Bridging the Editing Gap in LLMs: FineEdit for Precise and Targeted Text Modifications](https://mlsanity.com/paper/2502.13358/) (2025)
- [Automatic Metrics in Natural Language Generation: A Survey of Current Evaluation Practices](https://mlsanity.com/paper/2408.09169/) (2024)
- [Flesch or Fumble? Evaluating Readability Standard Alignment of Instruction-Tuned Language Models](https://mlsanity.com/paper/2309.05454/) (2023)
- [Learning to Paraphrase Sentences to Different Complexity Levels](https://mlsanity.com/paper/2308.02226/) (2023)
- [An LLM-Enhanced Adversarial Editing System for Lexical Simplification](https://mlsanity.com/paper/2402.14704/) (2024)
- [Progressive Document-level Text Simplification via Large Language Models](https://mlsanity.com/paper/2501.03857/) (2025)
- [A New Dataset and Empirical Study for Sentence Simplification in Chinese](https://mlsanity.com/paper/2306.04188/) (2023)
- [VTechAGP: An Academic-to-General-Audience Text Paraphrase Dataset and Benchmark Models](https://mlsanity.com/paper/2411.04825/) (2024)
- [A Structured Literature Review on Traditional Approaches in Current Natural Language Processing](https://mlsanity.com/paper/2505.12970/) (2025)

## Abstract (arXiv)

Randomly masking text spans in ordinary texts in the pre-training stage hardly allows models to acquire the ability to generate simple texts. It can hurt the performance of pre-trained models on text simplification tasks. In this paper, we propose a new continued pre-training strategy to teach the pre-trained model to generate simple texts. We continue pre-training BART, a representative model, to obtain SimpleBART. It consistently and significantly improves the results on lexical simplification, sentence simplification, and document-level simplification tasks over BART. At the end, we compare SimpleBART with several representative large language models (LLMs).