# Uniform Complexity for Text Generation

arXiv: https://arxiv.org/abs/2204.05185 · HTML: https://mlsanity.com/paper/2204.05185/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 86 references that are in the map.
- First posted on arXiv 11 Apr 2022.

## Builds on (most cited first, 24 of 24)

- [Hierarchical Neural Story Generation](https://mlsanity.com/paper/1805.04833/) (2018)
- [Plug and Play Language Models: A Simple Approach to Controlled Text Generation](https://mlsanity.com/paper/1912.02164/) (2019)
- [XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalization](https://mlsanity.com/paper/2003.11080/) (2020)
- [XCOPA: A Multilingual Dataset for Causal Commonsense Reasoning](https://mlsanity.com/paper/2005.00333/) (2020)
- [Why We Need New Evaluation Metrics for NLG](https://mlsanity.com/paper/1707.06875/) (2017)
- [Reformulating Unsupervised Style Transfer as Paraphrase Generation](https://mlsanity.com/paper/2010.05700/) (2020)
- [Do Long-Range Language Models Actually Use Long-Range Context?](https://mlsanity.com/paper/2109.09115/) (2021)
- [Do Massively Pretrained Language Models Make Better Storytellers?](https://mlsanity.com/paper/1909.10705/) (2019)
- [Continual Learning in Task-Oriented Dialogue Systems](https://mlsanity.com/paper/2012.15504/) (2020)
- [MEGATRON-CNTRL: Controllable Story Generation with External Knowledge Using Large-Scale Language Models](https://mlsanity.com/paper/2010.00840/) (2020)
- [UNION: An Unreferenced Metric for Evaluating Open-ended Story Generation](https://mlsanity.com/paper/2009.07602/) (2020)
- [STORIUM: A Dataset and Evaluation Platform for Machine-in-the-Loop Story Generation](https://mlsanity.com/paper/2010.01717/) (2020)
- [A Plug-and-Play Method for Controlled Text Generation](https://mlsanity.com/paper/2109.09707/) (2021)
- [Perturbation CheckLists for Evaluating NLG Evaluation Metrics](https://mlsanity.com/paper/2109.05771/) (2021)
- [Paraphrasing with Large Language Models](https://mlsanity.com/paper/1911.09661/) (2019)
- [Pushing on Text Readability Assessment: A Transformer Meets Handcrafted Linguistic Features](https://mlsanity.com/paper/2109.12258/) (2021)
- [Linguistic Features for Readability Assessment](https://mlsanity.com/paper/2006.00377/) (2020)
- [Controlling Text Complexity in Neural Machine Translation](https://mlsanity.com/paper/1911.00835/) (2019)
- [Curious Case of Language Generation Evaluation Metrics: A Cautionary Tale](https://mlsanity.com/paper/2010.13588/) (2020)
- [Decoding Methods for Neural Narrative Generation](https://mlsanity.com/paper/2010.07375/) (2020)
- [A Baseline Readability Model for Cebuano](https://mlsanity.com/paper/2203.17225/) (2022)
- [Changing the Mind of Transformers for Topically-Controllable Language Generation](https://mlsanity.com/paper/2103.15335/) (2021)
- [Readability-based Sentence Ranking for Evaluating Text Simplification](https://mlsanity.com/paper/1603.06009/) (2016)
- [Diverse Linguistic Features for Assessing Reading Difficulty of Educational Filipino Texts](https://mlsanity.com/paper/2108.00241/) (2021)

## Built on by (most cited first, 4 of 4)

- [Generating Summaries with Controllable Readability Levels](https://mlsanity.com/paper/2310.10623/) (2023)
- [Flesch or Fumble? Evaluating Readability Standard Alignment of Instruction-Tuned Language Models](https://mlsanity.com/paper/2309.05454/) (2023)
- [ReadCtrl: Personalizing text generation with readability-controlled instruction learning](https://mlsanity.com/paper/2406.09205/) (2024)
- [Free-text Rationale Generation under Readability Level Control](https://mlsanity.com/paper/2407.01384/) (2024)

## Abstract (arXiv)

Large language models (LLMs) have shown promising results in a wide array of generative NLP tasks, such as summarization and machine translation. In the context of narrative generation, however, existing models still do not capture factors that contribute to producing consistent text. For instance, it is logical that a piece of text or a story should be uniformly readable throughout and that this form of complexity should be controllable. As such, if the complexity of an input text prompt is rated first-grade reading level in the Flesch Reading Ease test, then the generated text continuing the plot should also be within this range of complexity. With this in mind, we introduce Uniform Complexity for Text Generation (UCTG), a new benchmark test which raises the challenge of making generative models observe uniform linguistic properties with respect to prompts. We experiment with over 150+ linguistically and cognitively motivated features for evaluating text complexity in humans and generative models. From our results, we find that models such as GPT-2 struggle to preserve the complexity of input prompts used in its generations, even if finetuned with professionally written texts.