mlsanityData to 14 Oct 2026

A Comparative Study of Controlled Text Generation Systems Using Level-Playing-Field Evaluation Principles

arXiv 2605.12395 ↗First posted 12 May 2026cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

Background: Many different approaches to controlled text generation (CTG) have been proposed over recent years, but it is difficult to get a clear picture of which approach performs best, because different datasets and evaluation methods are used in each case to assess the control achieved. Objectives: Our aim in the work reported in this paper is to develop an approach to evaluation that enables us to comparatively evaluate different CTG systems in a manner that is both informative and fair to the individual systems. Methods: We use a level-playing-field (LPF) approach to comparative evaluation where we (i) generate and process all system outputs in a standardised way, and (ii) apply a shared set of evaluation methods and datasets, selected based on those currently in use, in order to ensure fair evaluation. Results: When re-evaluated in this way, performance results for a representative set of current CTG systems differ substantially from originally reported results, in most cases for the worse. This highlights the importance of a shared standardised way of assessing controlled generation. Conclusions: The discrepancies revealed by LPF evaluation demonstrate the urgent need for standardised, reproducible evaluation practices in CTG. Our results suggest that without such practices, published performance claims may substantially misrepresent true system capabilities.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 37 of its 71 references that are in the map. First posted on arXiv 12 May 2026.

Builds on

37 of 71 references.

References · 37

  1. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  2. 2015Character-level Convolutional Networks for Text Classificationcited by 2,330
  3. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,984
  4. 2015A Diversity-Promoting Objective Function for Neural Conversation Modelscited by 1,077
  5. 2017SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluationcited by 1,001
  6. 2019Plug and Play Language Models: A Simple Approach to Controlled Text Generationcited by 799
  7. 2019CTRL: A Conditional Transformer Language Model for Controllable Generationcited by 762
  8. 2021DExperts: Decoding-Time Controlled Text Generation with Experts and Anti-Expertscited by 441
  9. 2020GeDi: Generative Discriminator Guided Sequence Generationcited by 378
  10. 2013One Billion Word Benchmark for Measuring Progress in Statistical Language Modelingcited by 308
  11. 2022Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Textcited by 155
  12. 2021NeuroLogic A*esque Decoding: Constrained Text Generation with Lookahead Heuristicscited by 154
25 more
  1. 2022Controllable Natural Language Generation with Contrastive Prefixescited by 75
  2. 2023Evaluating Large Language Models on Controlled Generation Taskscited by 73
  3. 2022Gradient-Based Constrained Sampling from Language Modelscited by 64
  4. 2021A Plug-and-Play Method for Controlled Text Generationcited by 56
  5. 2020Exploring Controllable Text Generation Techniquescited by 55
  6. 2019Neural data-to-text generation: A comparison between pipeline and end-to-end architecturescited by 51
  7. 2023ChatGPT vs Human-authored Text: Insights into Controllable Text Summarization and Sentence Style Transfercited by 50
  8. 2022On the Blind Spots of Model-Based Evaluation Metrics for Text Generationcited by 49
  9. 2022BERTScore is Unfair: On Social Bias in Language Model-Based Metrics for Text Generationcited by 48
  10. 2022Critic-Guided Decoding for Controlled Text Generationcited by 45
  11. 2018Sentence-Level Fluency Evaluation: References Help, But Can Be Spared!cited by 43
  12. 2022A Distributional Lens for Multi-Aspect Controllable Text Generationcited by 33
  13. 2023Controllable Mixed-Initiative Dialogue Generation through Promptingcited by 32
  14. 2020Plug-and-Play Conversational Modelscited by 30
  15. 2021Better than Average: Paired Evaluation of NLP Systemscited by 30
  16. 2022DisCup: Discriminator Cooperative Unlikelihood Prompt-tuning for Controllable Text Generationcited by 30
  17. 2022Controllable Text Generation via Probability Density Estimation in the Latent Spacecited by 27
  18. 2021Parallel Refinements for Lexically Constrained Text Generation with BARTcited by 24
  19. 2022Multi-Stage Prompting for Knowledgeable Dialogue Generationcited by 22
  20. 2021IGA : An Intent-Guided Authoring Assistantcited by 12
  21. 2023MacLaSa: Multi-Aspect Controllable Text Generation via Efficient Sampling from Compact Latent Spacecited by 12
  22. 2023Can Diffusion Model Achieve Better Performance in Text Generation? Bridging the Gap between Training and Inference!cited by 11
  23. 2023Focused Prefix Tuning for Controllable Text Generationcited by 9
  24. 2020Syntax-driven Iterative Expansion Language Models for Controllable Text Generationcited by 6
  25. 2022Nano: Nested Human-in-the-Loop Reward Learning for Few-shot Language Model Controlcited by 6

Built on by

Not cited yet.