# Assessing Quality of Experience in Natural Language Generation of German Text

arXiv: https://arxiv.org/abs/2608.18888 · HTML: https://mlsanity.com/paper/2608.18888/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 16 of its 48 references that are in the map.
- First posted on arXiv 19 Aug 2026.

## Builds on (most cited first, 16 of 16)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment](https://mlsanity.com/paper/2303.16634/) (2023)
- [Get To The Point: Summarization with Pointer-Generator Networks](https://mlsanity.com/paper/1704.04368/) (2017)
- [COMET: A Neural Framework for MT Evaluation](https://mlsanity.com/paper/2009.09025/) (2020)
- [SummEval: Re-evaluating Summarization Evaluation](https://mlsanity.com/paper/2007.12626/) (2020)
- [Experts, Errors, and Context: A Large-Scale Study of Human Evaluation for Machine Translation](https://mlsanity.com/paper/2104.14478/) (2021)
- [Why We Need New Evaluation Metrics for NLG](https://mlsanity.com/paper/1707.06875/) (2017)
- [Quality Assessment of In-the-Wild Videos](https://mlsanity.com/paper/1908.00375/) (2019)
- [German's Next Language Model](https://mlsanity.com/paper/2010.10906/) (2020)
- [Convolutional Self-Attention Networks](https://mlsanity.com/paper/1904.03107/) (2019)
- [Subjective Assessment of Text Complexity: A Dataset for German Language](https://mlsanity.com/paper/1904.07733/) (2019)
- [QualiSpeech: A Speech Quality Assessment Dataset with Natural Language Reasoning and Descriptions](https://mlsanity.com/paper/2503.20290/) (2025)
- [Text Summarization using Abstract Meaning Representation](https://mlsanity.com/paper/1706.01678/) (2017)
- [A Survey on Image Quality Assessment: Insights, Analysis, and Future Outlook](https://mlsanity.com/paper/2502.08540/) (2025)

## Abstract (arXiv)

The rapid advancement of Natural Language Generation (NLG) has made the reliable evaluation of generated text increasingly critical, as these systems, such as large language models (LLMs), are now widely deployed in real-world applications. However, traditional automatic metrics fail to capture the multifaceted nature of perceived quality. In this paper, we introduce TextQ-German, a novel dataset suite for human-centered evaluation of German NLG from a Quality of Experience (QoE) perspective, covering automatic text summarization and machine translation. Through crowdsourcing studies with German speakers, we collect human quality ratings and identify relevant perceptual quality dimensions for each task. We develop automatic QoE prediction models, including transformer-based, linguistic feature-based, and hybrid approaches. Hybrid models outperform pure transformer baselines in almost all experimental settings, while linguistic features alone can approach the performance of fine-tuned language models. The dataset is extended with LLM-generated outputs annotated with overall QoE scores. Final validation on held-out sets indicates generalization to unseen data. Our work contributes a publicly accessible resource for NLG evaluation and baselines for automatic QoE prediction, providing a foundation for developing NLG systems that better align with human quality perception.