mlsanityData to 7 Oct 2026

Teaching the Pre-trained Model to Generate Simple Texts for Text Simplification

arXiv 2305.12463 ↗First posted 21 May 2023cs.CL cs.AIcited by 10

Not on the map. Why?

Abstract (arXiv)

Randomly masking text spans in ordinary texts in the pre-training stage hardly allows models to acquire the ability to generate simple texts. It can hurt the performance of pre-trained models on text simplification tasks. In this paper, we propose a new continued pre-training strategy to teach the pre-trained model to generate simple texts. We continue pre-training BART, a representative model, to obtain SimpleBART. It consistently and significantly improves the results on lexical simplification, sentence simplification, and document-level simplification tasks over BART. At the end, we compare SimpleBART with several representative large language models (LLMs).

Key facts. Cited by 10 papers in the map (data through 7 Oct 2026). Builds on 25 of its 50 references that are in the map. First posted on arXiv 21 May 2023.

Builds on

25 of 50 references.

References · 25

  1. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  3. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  4. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  5. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  6. 2020Don't Stop Pretraining: Adapt Language Models to Domains and Taskscited by 1,548
  7. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,486
  8. 2019Text Summarization with Pretrained Encoderscited by 553
  9. 2017Sentence Simplification with Deep Reinforcement Learningcited by 107
  10. 2020Neural CRF Model for Sentence Alignment in Text Simplificationcited by 106
  11. 2018BLEU is Not Suitable for the Evaluation of Text Simplificationcited by 78
  12. 2019EditNTS: An Neural Programmer-Interpreter Model for Sentence Simplification through Explicit Editingcited by 63
13 more
  1. 2020Controllable Text Simplification with Explicit Paraphrasingcited by 53
  2. 2021Keep it Simple: Unsupervised Simplification of Multi-Paragraph Textcited by 45
  3. 2021Dict-BERT: Enhancing Language Model Pre-training with Dictionarycited by 38
  4. 2020Train No Evil: Selective Masking for Task-Guided Pre-Trainingcited by 36
  5. 2022Evaluating Factuality in Text Simplificationcited by 34
  6. 2021Text Simplification by Taggingcited by 27
  7. 2021Document-Level Text Simplification: Dataset, Criteria and Baselinecited by 27
  8. 2018A Word-Complexity Lexicon and A Neural Readability Ranking Model for Lexical Simplificationcited by 24
  9. 2020CompLex: A New Corpus for Lexical Complexity Prediction from Likert Scale Datacited by 24
  10. 2021SemEval-2021 Task 1: Lexical Complexity Predictioncited by 19
  11. 2021DEEP: DEnoising Entity Pre-training for Neural Machine Translationcited by 15
  12. 2020LSBert: A Simple Framework for Lexical Simplificationcited by 9
  13. 2023Exploiting Summarization Data to Help Text Simplificationcited by 5

Built on by

Cited by year

Citing papers per year: 2023: 4; 2024: 3; 2025: 3.

10 papers.

  1. 2023BLESS: Benchmarking Large Language Models on Sentence Simplificationcited by 34
  2. 2025Bridging the Editing Gap in LLMs: FineEdit for Precise and Targeted Text Modificationscited by 20
  3. 2024Automatic Metrics in Natural Language Generation: A Survey of Current Evaluation Practicescited by 19
  4. 2023Flesch or Fumble? Evaluating Readability Standard Alignment of Instruction-Tuned Language Modelscited by 17
  5. 2023Learning to Paraphrase Sentences to Different Complexity Levelscited by 11
  6. 2024An LLM-Enhanced Adversarial Editing System for Lexical Simplificationcited by 5
  7. 2025Progressive Document-level Text Simplification via Large Language Modelscited by 5
  8. 2023A New Dataset and Empirical Study for Sentence Simplification in Chinesecited by 4
  9. 2024VTechAGP: An Academic-to-General-Audience Text Paraphrase Dataset and Benchmark Modelscited by 3
  10. 2025A Structured Literature Review on Traditional Approaches in Current Natural Language Processingcited by 0