# Bounding the Capabilities of Large Language Models in Open Text Generation with Prompt Constraints

arXiv: https://arxiv.org/abs/2302.09185 · HTML: https://mlsanity.com/paper/2302.09185/

## Key facts

- Cited by 31 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 37 references that are in the map.
- First posted on arXiv 17 Feb 2023.

## Builds on (most cited first, 28 of 28)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [PaLM: Scaling Language Modeling with Pathways](https://mlsanity.com/paper/2204.02311/) (2022)
- [TruthfulQA: Measuring How Models Mimic Human Falsehoods](https://mlsanity.com/paper/2109.07958/) (2021)
- [TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension](https://mlsanity.com/paper/1705.03551/) (2017)
- [Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answering](https://mlsanity.com/paper/1809.02789/) (2018)
- [SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems](https://mlsanity.com/paper/1905.00537/) (2019)
- [Know What You Don't Know: Unanswerable Questions for SQuAD](https://mlsanity.com/paper/1806.03822/) (2018)
- [Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?](https://mlsanity.com/paper/2202.12837/) (2022)
- [RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Models](https://mlsanity.com/paper/2009.11462/) (2020)
- [LaMDA: Language Models for Dialog Applications](https://mlsanity.com/paper/2201.08239/) (2022)
- [Hierarchical Neural Story Generation](https://mlsanity.com/paper/1805.04833/) (2018)
- [GLM: General Language Model Pretraining with Autoregressive Blank Infilling](https://mlsanity.com/paper/2103.10360/) (2021)
- [Measuring Coding Challenge Competence With APPS](https://mlsanity.com/paper/2105.09938/) (2021)
- [DialoGPT: Large-Scale Generative Pre-training for Conversational Response Generation](https://mlsanity.com/paper/1911.00536/) (2019)
- [CTRL: A Conditional Transformer Language Model for Controllable Generation](https://mlsanity.com/paper/1909.05858/) (2019)
- [Prompt Programming for Large Language Models: Beyond the Few-Shot Paradigm](https://mlsanity.com/paper/2102.07350/) (2021)
- [A Corpus and Evaluation Framework for Deeper Understanding of Commonsense Stories](https://mlsanity.com/paper/1604.01696/) (2016)
- [BOLD: Dataset and Metrics for Measuring Biases in Open-Ended Language Generation](https://mlsanity.com/paper/2101.11718/) (2021)
- [GeDi: Generative Discriminator Guided Sequence Generation](https://mlsanity.com/paper/2009.06367/) (2020)
- [Can language models learn from explanations in context?](https://mlsanity.com/paper/2204.02329/) (2022)
- [Retrieving and Reading: A Comprehensive Survey on Open-domain Question Answering](https://mlsanity.com/paper/2101.00774/) (2021)
- [A Recipe For Arbitrary Text Style Transfer with Large Language Models](https://mlsanity.com/paper/2109.03910/) (2021)
- [STORIUM: A Dataset and Evaluation Platform for Machine-in-the-Loop Story Generation](https://mlsanity.com/paper/2010.01717/) (2020)
- [CoCon: A Self-Supervised Approach for Controlled Text Generation](https://mlsanity.com/paper/2006.03535/) (2020)

## Built on by (most cited first, 30 of 31)

- [Unveiling Security, Privacy, and Ethical Concerns of ChatGPT](https://mlsanity.com/paper/2307.14192/) (2023)
- [Understanding Information Storage and Transfer in Multi-modal Large Language Models](https://mlsanity.com/paper/2406.04236/) (2024)
- [Large Language Models as Tax Attorneys: A Case Study in Legal Capabilities Emergence](https://mlsanity.com/paper/2306.07075/) (2023)
- [SoK: Memorization in General-Purpose Large Language Models](https://mlsanity.com/paper/2310.18362/) (2023)
- [LLM Self-Correction with DeCRIM: Decompose, Critique, and Refine for Enhanced Following of Instructions with Multiple Constraints](https://mlsanity.com/paper/2410.06458/) (2024)
- [Measuring and Benchmarking Large Language Models' Capabilities to Generate Persuasive Language](https://mlsanity.com/paper/2406.17753/) (2024)
- [Adaptable Logical Control for Large Language Models](https://mlsanity.com/paper/2406.13892/) (2024)
- [SPRIG: Improving Large Language Model Performance by System Prompt Optimization](https://mlsanity.com/paper/2410.14826/) (2024)
- [Benchmarking Large Language Models on Controllable Generation under Diversified Instructions](https://mlsanity.com/paper/2401.00690/) (2024)
- [GROVE: A Retrieval-augmented Complex Story Generation Framework with A Forest of Evidence](https://mlsanity.com/paper/2310.05388/) (2023)
- [Subtle Biases Need Subtler Measures: Dual Metrics for Evaluating Representative and Affinity Bias in Large Language Models](https://mlsanity.com/paper/2405.14555/) (2024)
- [ELF-Gym: Evaluating Large Language Models Generated Features for Tabular Prediction](https://mlsanity.com/paper/2410.12865/) (2024)
- [CoDa: Constrained Generation based Data Augmentation for Low-Resource NLP](https://mlsanity.com/paper/2404.00415/) (2024)
- [Bandit-Based Prompt Design Strategy Selection Improves Prompt Optimizers](https://mlsanity.com/paper/2503.01163/) (2025)
- [ABEX: Data Augmentation for Low-Resource NLU via Expanding Abstract Descriptions](https://mlsanity.com/paper/2406.04286/) (2024)
- [Mitigating Bias in Locally Constrained Decoding via Tractable Proposals](https://mlsanity.com/paper/2606.01926/) (2026)
- [Developing Effective Educational Chatbots with ChatGPT prompts: Insights from Preliminary Tests in a Case Study on Social Media Literacy (with appendix)](https://mlsanity.com/paper/2306.10645/) (2023)
- [LLM can Achieve Self-Regulation via Hyperparameter Aware Generation](https://mlsanity.com/paper/2402.11251/) (2024)
- [A Large Language Model-Enabled Control Architecture for Dynamic Resource Capability Exploration in Multi-Agent Manufacturing Systems](https://mlsanity.com/paper/2505.22814/) (2025)
- [Say It My Way: Exploring Control in Conversational Visual Question Answering with Blind Users](https://mlsanity.com/paper/2602.16930/) (2026)
- [When Giant Language Brains Just Aren't Enough! Domain Pizzazz with Knowledge Sparkle Dust](https://mlsanity.com/paper/2305.07230/) (2023)
- [Self-Review Framework for Enhancing Instruction Following Capability of LLM](https://mlsanity.com/paper/2507.05598/) (2025)
- [LexChronos: An Agentic Framework for Structured Event Timeline Extraction in Indian Jurisprudence](https://mlsanity.com/paper/2603.01651/) (2026)
- [Evaluating, Understanding, and Improving Constrained Text Generation for Large Language Models](https://mlsanity.com/paper/2310.16343/) (2023)
- [Task-oriented Prompt Enhancement via Script Generation](https://mlsanity.com/paper/2409.16418/) (2024)
- [Automatic Prompt Generation and Grounding Object Detection for Zero-Shot Image Anomaly Detection](https://mlsanity.com/paper/2411.19220/) (2024)
- [Knowledge Conceptualization Impacts RAG Efficacy](https://mlsanity.com/paper/2507.09389/) (2025)
- [From Instruction to Output: The Role of Prompting in Modern NLG](https://mlsanity.com/paper/2602.11179/) (2026)
- [The Parser Already Knows: Lightweight Bias Correction in Constrained Decoding](https://mlsanity.com/paper/2608.10137/) (2026)
- [Do Influence Tactics Matter? Investigating Prompt Framing Effects in LLM Code Generation](https://mlsanity.com/paper/2608.11513/) (2026)

## Abstract (arXiv)

The limits of open-ended generative models are unclear, yet increasingly important. What causes them to succeed and what causes them to fail? In this paper, we take a prompt-centric approach to analyzing and bounding the abilities of open-ended generative models. We present a generic methodology of analysis with two challenging prompt constraint types: structural and stylistic. These constraint types are categorized into a set of well-defined constraints that are analyzable by a single prompt. We then systematically create a diverse set of simple, natural, and useful prompts to robustly analyze each individual constraint. Using the GPT-3 text-davinci-002 model as a case study, we generate outputs from our collection of prompts and analyze the model's generative failures. We also show the generalizability of our proposed method on other large models like BLOOM and OPT. Our results and our in-context mitigation strategies reveal open challenges for future research. We have publicly released our code at https://github.com/SALT-NLP/Bound-Cap-LLM.