# Language Models for German Text Simplification: Overcoming Parallel Data Scarcity through Style-specific Pre-training

arXiv: https://arxiv.org/abs/2305.12908 · HTML: https://mlsanity.com/paper/2305.12908/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 42 references that are in the map.
- First posted on arXiv 22 May 2023.

## Builds on (most cited first, 15 of 15)

- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [PaLM: Scaling Language Modeling with Pathways](https://mlsanity.com/paper/2204.02311/) (2022)
- [Don't Stop Pretraining: Adapt Language Models to Domains and Tasks](https://mlsanity.com/paper/2004.10964/) (2020)
- [Multilingual Denoising Pre-training for Neural Machine Translation](https://mlsanity.com/paper/2001.08210/) (2020)
- [An Empirical Investigation of Catastrophic Forgetting in Gradient-Based Neural Networks](https://mlsanity.com/paper/1312.6211/) (2013)
- [A Contrastive Framework for Neural Text Generation](https://mlsanity.com/paper/2202.06417/) (2022)
- [mGPT: Few-Shot Learners Go Multilingual](https://mlsanity.com/paper/2204.07580/) (2022)
- [WECHSEL: Effective initialization of subword embeddings for cross-lingual transfer of monolingual language models](https://mlsanity.com/paper/2112.06598/) (2021)
- [Cross-Attention is All You Need: Adapting Pretrained Transformers for Machine Translation](https://mlsanity.com/paper/2104.08771/) (2021)
- [What Kind of Language Is Hard to Language-Model?](https://mlsanity.com/paper/1906.04726/) (2019)
- [Klexikon: A German Dataset for Joint Summarization and Simplification](https://mlsanity.com/paper/2201.07198/) (2022)
- [GPT-based Generation for Classical Chinese Poetry](https://mlsanity.com/paper/1907.00151/) (2019)
- [A New Aligned Simple German Corpus](https://mlsanity.com/paper/2209.01106/) (2022)
- [GPT-D: Inducing Dementia-related Linguistic Anomalies by Deliberate Degradation of Artificial Neural Language Models](https://mlsanity.com/paper/2203.13397/) (2022)
- [SimpleBERT: A Pre-trained Model That Learns to Generate Simple Words](https://mlsanity.com/paper/2204.07779/) (2022)

## Built on by (most cited first, 15 of 15)

- [mEdIT: Multilingual Text Editing via Instruction Tuning](https://mlsanity.com/paper/2402.16472/) (2024)
- [German Text Simplification: Finetuning Large Language Models with Semi-Synthetic Data](https://mlsanity.com/paper/2402.10675/) (2024)
- [Using ChatGPT as a CAT tool in Easy Language translation](https://mlsanity.com/paper/2308.11563/) (2023)
- [Data and Approaches for German Text simplification -- towards an Accessibility-enhanced Communication](https://mlsanity.com/paper/2312.09966/) (2023)
- [Images Speak Volumes: User-Centric Assessment of Image Generation for Accessible Communication](https://mlsanity.com/paper/2410.03430/) (2024)
- [German4All -- A Dataset and Model for Readability-Controlled Paraphrasing in German](https://mlsanity.com/paper/2508.17973/) (2025)
- [Facilitating Cognitive Accessibility with LLMs: A Multi-Task Approach to Easy-to-Read Text Generation](https://mlsanity.com/paper/2510.00662/) (2025)
- [Edit-Constrained Decoding for Sentence Simplification](https://mlsanity.com/paper/2409.19247/) (2024)
- [SimplifyMyText: An LLM-Based System for Inclusive Plain Language Text Simplification](https://mlsanity.com/paper/2504.14223/) (2025)
- [Exploring Automatic Text Simplification of German Narrative Documents](https://mlsanity.com/paper/2312.09907/) (2023)
- [EASSE-DE: Easier Automatic Sentence Simplification Evaluation for German](https://mlsanity.com/paper/2404.03563/) (2024)
- [A Structured Literature Review on Traditional Approaches in Current Natural Language Processing](https://mlsanity.com/paper/2505.12970/) (2025)
- [Evaluating the Effectiveness of Direct Preference Optimization for Personalizing German Automatic Text Simplifications for Persons with Intellectual Disabilities](https://mlsanity.com/paper/2507.01479/) (2025)
- [DETECT: Determining Ease and Textual Clarity of German Text Simplifications](https://mlsanity.com/paper/2510.22212/) (2025)
- [Profiling German Text Simplification with Interpretable Model-Fingerprints](https://mlsanity.com/paper/2601.13050/) (2026)

## Abstract (arXiv)

Automatic text simplification systems help to reduce textual information barriers on the internet. However, for languages other than English, only few parallel data to train these systems exists. We propose a two-step approach to overcome this data scarcity issue. First, we fine-tuned language models on a corpus of German Easy Language, a specific style of German. Then, we used these models as decoders in a sequence-to-sequence simplification task. We show that the language models adapt to the style characteristics of Easy Language and output more accessible texts. Moreover, with the style-specific pre-training, we reduced the number of trainable parameters in text simplification models. Hence, less parallel data is sufficient for training. Our results indicate that pre-training on unaligned data can reduce the required parallel data while improving the performance on downstream tasks.