mlsanityData to 14 Oct 2026

COLLIE: Systematic Construction of Constrained Text Generation Tasks

arXiv 2307.08689 ↗First posted 17 Jul 2023cs.CL cs.AI cs.LGcited by 70

Not on the map. Why?

Abstract (arXiv)

Text generation under constraints have seen increasing interests in natural language processing, especially with the rapidly improving capabilities of large language models. However, existing benchmarks for constrained generation usually focus on fixed constraint types (e.g.,generate a sentence containing certain words) that have proved to be easy for state-of-the-art models like GPT-4. We present COLLIE, a grammar-based framework that allows the specification of rich, compositional constraints with diverse generation levels (word, sentence, paragraph, passage) and modeling challenges (e.g.,language understanding, logical reasoning, counting, semantic planning). We also develop tools for automatic extraction of task instances given a constraint structure and a raw text corpus. Using COLLIE, we compile the COLLIE-v1 dataset with 2080 instances comprising 13 constraint structures. We perform systematic experiments across five state-of-the-art instruction-tuned language models and analyze their performances to reveal shortcomings. COLLIE is designed to be extensible and lightweight, and we hope the community finds it useful to develop more complex constraints and evaluations in the future.

Key facts. Cited by 70 papers in the map (data through 14 Oct 2026). Builds on 28 of its 45 references that are in the map. First posted on arXiv 17 Jul 2023.

Builds on

28 of 45 references.

References · 28

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2022Training language models to follow instructions with human feedbackcited by 15,971
  4. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
  5. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  6. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  7. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  8. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,182
  9. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,998
  10. 2016CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoningcited by 1,457
  11. 2023PaLM 2 Technical Reportcited by 1,162
  12. 2022Diffusion-LM Improves Controllable Text Generationcited by 1,027
16 more
  1. 2022Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Taskscited by 868
  2. 2019Plug and Play Language Models: A Simple Approach to Controlled Text Generationcited by 799
  3. 2019CTRL: A Conditional Transformer Language Model for Controllable Generationcited by 762
  4. 2021Datasets: A Community Library for Natural Language Processingcited by 530
  5. 2020GeDi: Generative Discriminator Guided Sequence Generationcited by 378
  6. 2015Towards AI-Complete Question Answering: A Set of Prerequisite Toy Taskscited by 314
  7. 2017Toward Controlled Generation of Textcited by 272
  8. 2018BabyAI: A Platform to Study the Sample Efficiency of Grounded Language Learningcited by 244
  9. 2017Lexically Constrained Decoding for Sequence Generation Using Grid Beam Searchcited by 237
  10. 2021NeuroLogic A*esque Decoding: Constrained Text Generation with Lookahead Heuristicscited by 154
  11. 2020NeuroLogic Decoding: (Un)supervised Neural Text Generation with Predicate Logic Constraintscited by 149
  12. 2020A Benchmark for Systematic Generalization in Grounded Language Understandingcited by 108
  13. 2019Training Neural Machine Translation To Apply Terminology Constraintscited by 73
  14. 2018Neural Machine Translation Decoding with Terminology Constraintscited by 36
  15. 2022Controllable Text Generation with Language Constraintscited by 18
  16. 2021Quantifying Adaptability in Pre-trained Language Models with 500 Taskscited by 13

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 5; 2024: 16; 2025: 26; 2026: 23.

70 papers.

Sort:

By area:

  1. 2023Evaluating Large Language Models at Evaluating Instruction Followingcited by 302
  2. 2025Reinforcement Learning with Rubric Anchorscited by 113
  3. 2023Branch-Solve-Merge Improves Large Language Model Evaluation and Generationcited by 106
  4. 2024The BiGGen Bench: A Principled Benchmark for Fine-grained Evaluation of Language Models with Language Modelscited by 93
  5. 2024Roleplay-doh: Enabling Domain-Experts to Create LLM-simulated Patients via Eliciting and Adhering to Principlescited by 73
  6. 2025JSONSchemaBench: A Rigorous Benchmark of Structured Outputs for Language Modelscited by 71
  7. 2023Can Large Language Models Understand Real-World Complex Instructions?cited by 69
  8. 2024CFBench: A Comprehensive Constraints-Following Benchmark for LLMscited by 67
  9. 2024A Comparative Study on Reasoning Patterns of OpenAI's o1 Modelcited by 63
  10. 2025LongCat-Flash Technical Reportcited by 58
  11. 2026Seed1.8 Model Card: Towards Generalized Real-World Agencycited by 50
  12. 2024LLM Self-Correction with DeCRIM: Decompose, Critique, and Refine for Enhanced Following of Instructions with Multiple Constraintscited by 35
38 more
  1. 2024From Complex to Simple: Enhancing Multi-Constraint Complex Instruction Following Ability of Large Language Modelscited by 27
  2. 2025LongCat-Flash-Omni Technical Reportcited by 26
  3. 2024CS4: Measuring the Creativity of Large Language Models Automatically by Controlling the Number of Story-Writing Constraintscited by 24
  4. 2024Can Models Learn Skill Composition from Examples?cited by 17
  5. 2025Constrained Discrete Diffusioncited by 17
  6. 2025From Code Foundation Models to Agents and Applications: A Comprehensive Survey and Practical Guide to Code Intelligencecited by 15
  7. 2025Self-Steering Language Modelscited by 13
  8. 2023FollowEval: A Multi-Dimensional Benchmark for Assessing the Instruction-Following Capability of Large Language Modelscited by 12
  9. 2024KIWI: A Dataset of Knowledge-Intensive Writing Instructions for Answering Research Questionscited by 11
  10. 2025WildIFEval: Instruction Following in the Wildcited by 11
  11. 2025EvalAgent: Discovering Implicit Evaluation Criteria from the Webcited by 11
  12. 2024BenchAgents: Multi-Agent Systems for Structured Benchmark Creationcited by 9
  13. 2024Combining Constraint Programming Reasoning with Large Language Model Predictionscited by 8
  14. 2024Controllable Text Generation in the Instruction-Tuning Eracited by 7
  15. 2025Addressing Pitfalls in the Evaluation of Uncertainty Estimation Methods for Natural Language Generationcited by 7
  16. 2025The Ouroboros of Benchmarking: Reasoning Evaluation in an Era of Saturationcited by 7
  17. 2025Revisiting the Reliability of Language Models in Instruction-Followingcited by 7
  18. 2026CCTU: A Benchmark for Tool Use under Complex Constraintscited by 7
  19. 2025Instructions are all you need: Self-supervised Reinforcement Learning for Instruction Followingcited by 6
  20. 2026WavBench: Benchmarking Reasoning, Colloquialism, and Paralinguistics for End-to-End Spoken Dialogue Modelscited by 6
  21. 2024Preference-Guided Reflective Sampling for Aligning Language Modelscited by 4
  22. 2025Assessing Robustness to Spurious Correlations in Post-Training Language Modelscited by 4
  23. 2025SOP-Maze: Evaluating Large Language Models on Complicated Business Standard Operating Procedurescited by 4
  24. 2025Code-enabled language models can outperform reasoning models on diverse taskscited by 4
  25. 2026Neuro-Symbolic Verification on Instruction Following of LLMscited by 4
  26. 2024Train & Constrain: Phonologically Informed Tongue-Twister Generation from Topics and Paraphrasescited by 3
  27. 2024Intertwining CP and NLP: The Generation of Unreasonably Constrained Sentencescited by 3
  28. 2025Meeseeks: A Feedback-Driven, Iterative Self-Correction Benchmark evaluating LLMs' Instruction Following Capabilitycited by 3
  29. 2025Beyond the Trade-off: Self-Supervised Reinforcement Learning for Reasoning Models' Instruction Followingcited by 3
  30. 2025Large-Scale Constraint Generation -- Can LLMs Parse Hundreds of Constraints?cited by 3
  31. 2024ExpressivityBench: Can LLMs Communicate Implicitly?cited by 2
  32. 2025Large Language Model Meets Constraint Propagationcited by 2
  33. 2026FireBench: Evaluating Instruction Following in Enterprise and API-Driven LLM Applicationscited by 2
  34. 2026MM-CondChain: A Programmatically Verified Benchmark for Visually Grounded Deep Compositional Reasoningcited by 2
  35. 2026SAGE: A Service Agent Graph-guided Evaluation Benchmarkcited by 2
  36. 2025LCTG Bench: LLM Controlled Text Generation Benchmarkcited by 1
  37. 2025UltraGen: Extremely Fine-grained Controllable Generation via Attribute Reconstruction and Global Preference Optimizationcited by 1
  38. 2025Learning to Align: Addressing Character Frequency Distribution Shifts in Handwritten Text Recognitioncited by 1

50 most cited of 70 All 70 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Benchmarking Large Language Models on Controllable Generation under Diversified Instructions
  2. 2022COLD Decoding: Energy-based Constrained Text Generation with Langevin Dynamics
  3. 2025A Survey on Parallel Text Generation: From Parallel Decoding to Diffusion Language Models
  4. 2024Guiding LLMs The Right Way: Fast, Non-Invasive Constrained Generation
  5. 2026Understanding Axes of Difficulty For Long Context Tasks Via PredicateLongBench
  6. 2024Retrieval is Accurate Generation
  7. 2023SituatedGen: Incorporating Geographical and Temporal Contexts into Generative Commonsense Reasoning
  8. 2024From Instructions to Constraints: Language Model Alignment with Automatic Constraint Verification