# Evaluating Large Language Models on Controlled Generation Tasks

arXiv: https://arxiv.org/abs/2310.14542 · HTML: https://mlsanity.com/paper/2310.14542/

## Key facts

- Cited by 73 papers in the map (data through 14 Oct 2026).
- Builds on 43 of its 65 references that are in the map.
- First posted on arXiv 23 Oct 2023.

## Builds on (most cited first, 30 of 43)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [The Curious Case of Neural Text Degeneration](https://mlsanity.com/paper/1904.09751/) (2019)
- [SimCSE: Simple Contrastive Learning of Sentence Embeddings](https://mlsanity.com/paper/2104.08821/) (2021)
- [Hierarchical Neural Story Generation](https://mlsanity.com/paper/1805.04833/) (2018)
- [Diffusion-LM Improves Controllable Text Generation](https://mlsanity.com/paper/2205.14217/) (2022)
- [Contrastive Decoding: Open-ended Text Generation as Optimization](https://mlsanity.com/paper/2210.15097/) (2022)
- [Towards Robust Interpretability with Self-Explaining Neural Networks](https://mlsanity.com/paper/1806.07538/) (2018)
- [Neural Text Generation with Unlikelihood Training](https://mlsanity.com/paper/1908.04319/) (2019)
- [Is ChatGPT a General-Purpose Natural Language Processing Task Solver?](https://mlsanity.com/paper/2302.06476/) (2023)
- [MAUVE: Measuring the Gap Between Neural Text and Human Text using Divergence Frontiers](https://mlsanity.com/paper/2102.01454/) (2021)
- [A Corpus and Evaluation Framework for Deeper Understanding of Commonsense Stories](https://mlsanity.com/paper/1604.01696/) (2016)
- [Explain Yourself! Leveraging Language Models for Commonsense Reasoning](https://mlsanity.com/paper/1906.02361/) (2019)
- [MEGA: Multilingual Evaluation of Generative AI](https://mlsanity.com/paper/2303.12528/) (2023)
- [Adversarial Example Generation with Syntactically Controlled Paraphrase Networks](https://mlsanity.com/paper/1804.06059/) (2018)
- [A Contrastive Framework for Neural Text Generation](https://mlsanity.com/paper/2202.06417/) (2022)
- [Fast Lexically Constrained Decoding with Dynamic Beam Allocation for Neural Machine Translation](https://mlsanity.com/paper/1804.06609/) (2018)
- [Is ChatGPT A Good Translator? Yes With GPT-4 As The Engine](https://mlsanity.com/paper/2301.08745/) (2023)
- [COLD Decoding: Energy-based Constrained Text Generation with Langevin Dynamics](https://mlsanity.com/paper/2202.11705/) (2022)
- [NeuroLogic Decoding: (Un)supervised Neural Text Generation with Predicate Logic Constraints](https://mlsanity.com/paper/2010.12884/) (2020)
- [The Multilingual Amazon Reviews Corpus](https://mlsanity.com/paper/2010.02573/) (2020)
- [Truncation Sampling as Language Model Desmoothing](https://mlsanity.com/paper/2210.15191/) (2022)
- [A Systematic Study and Comprehensive Evaluation of ChatGPT on Benchmark Datasets](https://mlsanity.com/paper/2305.18486/) (2023)
- [Teaching Algorithmic Reasoning via In-context Learning](https://mlsanity.com/paper/2211.09066/) (2022)
- [Guided Open Vocabulary Image Captioning with Constrained Beam Search](https://mlsanity.com/paper/1612.00576/) (2016)
- [Controlled Text Generation with Natural Language Instructions](https://mlsanity.com/paper/2304.14293/) (2023)
- [Exploring the Feasibility of ChatGPT for Event Extraction](https://mlsanity.com/paper/2303.03836/) (2023)
- [Do Models Really Learn to Follow Instructions? An Empirical Study of Instruction Tuning](https://mlsanity.com/paper/2305.11383/) (2023)

## Built on by (most cited first, 30 of 73)

- [Instruction-Following Evaluation for Large Language Models](https://mlsanity.com/paper/2311.07911/) (2023)
- [Generalizing Verifiable Instruction Following](https://mlsanity.com/paper/2507.02833/) (2025)
- [Fact-Checking the Output of Large Language Models via Token-Level Uncertainty Quantification](https://mlsanity.com/paper/2403.04696/) (2024)
- [Improving Instruction-Following in Language Models through Activation Steering](https://mlsanity.com/paper/2410.12877/) (2024)
- [Multi-IF: Benchmarking LLMs on Multi-Turn and Multilingual Instructions Following](https://mlsanity.com/paper/2410.15553/) (2024)
- [Prompt Engineering a Prompt Engineer](https://mlsanity.com/paper/2311.05661/) (2023)
- [LLM Self-Correction with DeCRIM: Decompose, Critique, and Refine for Enhanced Following of Instructions with Multiple Constraints](https://mlsanity.com/paper/2410.06458/) (2024)
- [SAIF: A Sparse Autoencoder Framework for Interpreting and Steering Instruction Following of Language Models](https://mlsanity.com/paper/2502.11356/) (2025)
- [Adaptable Logical Control for Large Language Models](https://mlsanity.com/paper/2406.13892/) (2024)
- [Conifer: Improving Complex Constrained Instruction-Following Ability of Large Language Models](https://mlsanity.com/paper/2404.02823/) (2024)
- [From Complex to Simple: Enhancing Multi-Constraint Complex Instruction Following Ability of Large Language Models](https://mlsanity.com/paper/2404.15846/) (2024)
- [RuleArena: A Benchmark for Rule-Guided Reasoning with LLMs in Real-World Scenarios](https://mlsanity.com/paper/2412.08972/) (2024)
- [Spotlight Your Instructions: Instruction-following with Dynamic Attention Steering](https://mlsanity.com/paper/2505.12025/) (2025)
- [The SIFo Benchmark: Investigating the Sequential Instruction Following Ability of Large Language Models](https://mlsanity.com/paper/2406.19999/) (2024)
- [Taxonomy of User Needs and Actions](https://mlsanity.com/paper/2510.06124/) (2025)
- [Has this Fact been Edited? Detecting Knowledge Edits in Language Models](https://mlsanity.com/paper/2405.02765/) (2024)
- [Self-Steering Language Models](https://mlsanity.com/paper/2504.07081/) (2025)
- [FollowEval: A Multi-Dimensional Benchmark for Assessing the Instruction-Following Capability of Large Language Models](https://mlsanity.com/paper/2311.09829/) (2023)
- [LangGPT: Rethinking Structured Reusable Prompt Design Framework for LLMs from the Programming Language](https://mlsanity.com/paper/2402.16929/) (2024)
- [Evaluating Human Alignment and Model Faithfulness of LLM Rationale](https://mlsanity.com/paper/2407.00219/) (2024)
- [KIWI: A Dataset of Knowledge-Intensive Writing Instructions for Answering Research Questions](https://mlsanity.com/paper/2403.03866/) (2024)
- [Controllable Generation via Locally Constrained Resampling](https://mlsanity.com/paper/2410.13111/) (2024)
- [WildIFEval: Instruction Following in the Wild](https://mlsanity.com/paper/2503.06573/) (2025)
- [AI with Emotions: Exploring Emotional Expressions in Large Language Models](https://mlsanity.com/paper/2504.14706/) (2025)
- [Insertion Language Models: Sequence Generation with Arbitrary-Position Insertions](https://mlsanity.com/paper/2505.05755/) (2025)
- [ReadCtrl: Personalizing text generation with readability-controlled instruction learning](https://mlsanity.com/paper/2406.09205/) (2024)
- [Generalists vs. Specialists: Evaluating LLMs on Highly-Constrained Biophysical Sequence Optimization Tasks](https://mlsanity.com/paper/2410.22296/) (2024)
- [Zero-Shot Strategies for Length-Controllable Summarization](https://mlsanity.com/paper/2501.00233/) (2024)
- [ConInstruct: Evaluating Large Language Models on Conflict Detection and Resolution in Instructions](https://mlsanity.com/paper/2511.14342/) (2025)
- [Grasping the Essentials: Tailoring Large Language Models for Zero-Shot Relation Extraction](https://mlsanity.com/paper/2402.11142/) (2024)

## Abstract (arXiv)

While recent studies have looked into the abilities of large language models in various benchmark tasks, including question generation, reading comprehension, multilingual and etc, there have been few studies looking into the controllability of large language models on generation tasks. We present an extensive analysis of various benchmarks including a sentence planning benchmark with different granularities. After comparing large language models against state-of-the-start finetuned smaller models, we present a spectrum showing large language models falling behind, are comparable, or exceed the ability of smaller models. We conclude that **large language models struggle at meeting fine-grained hard constraints**.