mlsanityData to 7 Oct 2026

Evaluating Large Language Models on Controlled Generation Tasks

arXiv 2310.14542 ↗First posted 23 Oct 2023cs.CLcited by 73

Not on the map. Why?

Abstract (arXiv)

While recent studies have looked into the abilities of large language models in various benchmark tasks, including question generation, reading comprehension, multilingual and etc, there have been few studies looking into the controllability of large language models on generation tasks. We present an extensive analysis of various benchmarks including a sentence planning benchmark with different granularities. After comparing large language models against state-of-the-start finetuned smaller models, we present a spectrum showing large language models falling behind, are comparable, or exceed the ability of smaller models. We conclude that **large language models struggle at meeting fine-grained hard constraints**.

Key facts. Cited by 73 papers in the map (data through 7 Oct 2026). Builds on 43 of its 65 references that are in the map. First posted on arXiv 23 Oct 2023.

Builds on

43 of 65 references.

References · 43

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  5. 2019The Curious Case of Neural Text Degenerationcited by 2,881
  6. 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,168
  7. 2018Hierarchical Neural Story Generationcited by 1,182
  8. 2022Diffusion-LM Improves Controllable Text Generationcited by 1,006
  9. 2022Contrastive Decoding: Open-ended Text Generation as Optimizationcited by 598
  10. 2018Towards Robust Interpretability with Self-Explaining Neural Networkscited by 505
  11. 2019Neural Text Generation with Unlikelihood Trainingcited by 489
  12. 2023Is ChatGPT a General-Purpose Natural Language Processing Task Solver?cited by 486
31 more
  1. 2021MAUVE: Measuring the Gap Between Neural Text and Human Text using Divergence Frontierscited by 466
  2. 2016A Corpus and Evaluation Framework for Deeper Understanding of Commonsense Storiescited by 413
  3. 2019Explain Yourself! Leveraging Language Models for Commonsense Reasoningcited by 367
  4. 2023MEGA: Multilingual Evaluation of Generative AIcited by 291
  5. 2018Adversarial Example Generation with Syntactically Controlled Paraphrase Networkscited by 282
  6. 2022A Contrastive Framework for Neural Text Generationcited by 224
  7. 2018Fast Lexically Constrained Decoding with Dynamic Beam Allocation for Neural Machine Translationcited by 181
  8. 2023Is ChatGPT A Good Translator? Yes With GPT-4 As The Enginecited by 172
  9. 2022COLD Decoding: Energy-based Constrained Text Generation with Langevin Dynamicscited by 159
  10. 2020NeuroLogic Decoding: (Un)supervised Neural Text Generation with Predicate Logic Constraintscited by 146
  11. 2020The Multilingual Amazon Reviews Corpuscited by 142
  12. 2023A Systematic Study and Comprehensive Evaluation of ChatGPT on Benchmark Datasetscited by 123
  13. 2022Truncation Sampling as Language Model Desmoothingcited by 121
  14. 2022Teaching Algorithmic Reasoning via In-context Learningcited by 111
  15. 2016Guided Open Vocabulary Image Captioning with Constrained Beam Searchcited by 105
  16. 2023Controlled Text Generation with Natural Language Instructionscited by 87
  17. 2023Exploring the Feasibility of ChatGPT for Event Extractioncited by 76
  18. 2023Do Models Really Learn to Follow Instructions? An Empirical Study of Instruction Tuningcited by 48
  19. 2020Syntax-guided Controlled Generation of Paraphrasescited by 47
  20. 2021Generating Syntactically Controlled Paraphrases without Using Annotated Parallel Pairscited by 45
  21. 2020Paraphrase Augmented Task-Oriented Dialog Generationcited by 43
  22. 2022Controllable Text Generation with Neurally-Decomposed Oraclecited by 41
  23. 2023Did You Read the Instructions? Rethinking the Effectiveness of Task Definitions in Instruction Learningcited by 40
  24. 2020Controlling Style in Generated Dialoguecited by 38
  25. 2019A Multi-Task Approach for Disentangling Syntax and Semantics in Sentence Representationscited by 37
  26. 2022M2D2: A Massively Multi-domain Language Modeling Datasetcited by 26
  27. 2022Zero-shot Sonnet Generation with Discourse-level Planning and Aesthetics Featurescited by 24
  28. 2022Language model acceptability judgements are not always robust to contextcited by 22
  29. 2022An Empirical Study On Contrastive Search And Contrastive Decoding For Open-ended Text Generationcited by 14
  30. 2023Look-back Decoding for Open-Ended Text Generationcited by 10
  31. 2022Towards Robust NLG Bias Evaluation with Syntactically-diverse Promptscited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 4; 2024: 38; 2025: 22; 2026: 9.

73 papers.

Sort:

By area:

  1. 2023Instruction-Following Evaluation for Large Language Modelscited by 1,143
  2. 2025Generalizing Verifiable Instruction Followingcited by 178
  3. 2024Fact-Checking the Output of Large Language Models via Token-Level Uncertainty Quantificationcited by 159
  4. 2024Improving Instruction-Following in Language Models through Activation Steeringcited by 149
  5. 2024Multi-IF: Benchmarking LLMs on Multi-Turn and Multilingual Instructions Followingcited by 94
  6. 2023Prompt Engineering a Prompt Engineercited by 53
  7. 2024LLM Self-Correction with DeCRIM: Decompose, Critique, and Refine for Enhanced Following of Instructions with Multiple Constraintscited by 34
  8. 2025SAIF: A Sparse Autoencoder Framework for Interpreting and Steering Instruction Following of Language Modelscited by 32
  9. 2024Adaptable Logical Control for Large Language Modelscited by 30
  10. 2024From Complex to Simple: Enhancing Multi-Constraint Complex Instruction Following Ability of Large Language Modelscited by 27
  11. 2024Conifer: Improving Complex Constrained Instruction-Following Ability of Large Language Modelscited by 26
  12. 2024RuleArena: A Benchmark for Rule-Guided Reasoning with LLMs in Real-World Scenarioscited by 26
38 more
  1. 2024The SIFo Benchmark: Investigating the Sequential Instruction Following Ability of Large Language Modelscited by 18
  2. 2025Spotlight Your Instructions: Instruction-following with Dynamic Attention Steeringcited by 18
  3. 2024Has this Fact been Edited? Detecting Knowledge Edits in Language Modelscited by 14
  4. 2025Self-Steering Language Modelscited by 13
  5. 2025Taxonomy of User Needs and Actionscited by 13
  6. 2023FollowEval: A Multi-Dimensional Benchmark for Assessing the Instruction-Following Capability of Large Language Modelscited by 12
  7. 2024LangGPT: Rethinking Structured Reusable Prompt Design Framework for LLMs from the Programming Languagecited by 12
  8. 2024Evaluating Human Alignment and Model Faithfulness of LLM Rationalecited by 12
  9. 2024KIWI: A Dataset of Knowledge-Intensive Writing Instructions for Answering Research Questionscited by 11
  10. 2024Controllable Generation via Locally Constrained Resamplingcited by 11
  11. 2025Insertion Language Models: Sequence Generation with Arbitrary-Position Insertionscited by 11
  12. 2024ReadCtrl: Personalizing text generation with readability-controlled instruction learningcited by 10
  13. 2024Zero-Shot Strategies for Length-Controllable Summarizationcited by 10
  14. 2025WildIFEval: Instruction Following in the Wildcited by 10
  15. 2025AI with Emotions: Exploring Emotional Expressions in Large Language Modelscited by 10
  16. 2024Grasping the Essentials: Tailoring Large Language Models for Zero-Shot Relation Extractioncited by 9
  17. 2024PositionID: LLMs can Control Lengths, Copy and Paste with Explicit Positional Awarenesscited by 9
  18. 2025ConInstruct: Evaluating Large Language Models on Conflict Detection and Resolution in Instructionscited by 9
  19. 2024From Instructions to Constraints: Language Model Alignment with Automatic Constraint Verificationcited by 8
  20. 2024ABEX: Data Augmentation for Low-Resource NLU via Expanding Abstract Descriptionscited by 8
  21. 2024Generalists vs. Specialists: Evaluating LLMs on Highly-Constrained Biophysical Sequence Optimization Taskscited by 8
  22. 2024PhonologyBench: Evaluating Phonological Skills of Large Language Modelscited by 7
  23. 2024Controllable Text Generation in the Instruction-Tuning Eracited by 7
  24. 2024Creativity in AI: Progresses and Challengescited by 7
  25. 2025Revisiting the Reliability of Language Models in Instruction-Followingcited by 6
  26. 2024Control Large Language Models via Divide and Conquercited by 5
  27. 2025Collaborative Document Editing with Multiple Users and AI Agentscited by 5
  28. 2025Semantic Probabilistic Control of Language Modelscited by 4
  29. 2024V-RECS, a Low-Cost LLM4VIS Recommender with Explanations, Captioning and Suggestionscited by 3
  30. 2025LLM-driven Constrained Copy Generation through Iterative Refinementcited by 3
  31. 2025Syntactic Control of Language Models by Posterior Inferencecited by 3
  32. 2025Large-Scale Constraint Generation -- Can LLMs Parse Hundreds of Constraints?cited by 3
  33. 2026Controlling Reading Ease with Gaze-Guided Text Generationcited by 3
  34. 2024Fine-grained Controllable Text Generation through In-context Learning with Feedbackcited by 2
  35. 2024Evaluation of Instruction-Following Ability for Large Language Models on Story-Ending Generationcited by 2
  36. 2024SpeciaLex: A Benchmark for In-Context Specialized Lexicon Learningcited by 2
  37. 2024REFFLY: Melody-Constrained Lyrics Editing Modelcited by 2
  38. 2025Grammar Control in Dialogue Response Generation for Language Learning Chatbotscited by 2

50 most cited of 73 All 73 (JSON)