# "We Need Structured Output": Towards User-centered Constraints on Large Language Model Output

arXiv: https://arxiv.org/abs/2404.07362 · HTML: https://mlsanity.com/paper/2404.07362/

## Key facts

- Cited by 46 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 10 Apr 2024.

## Built on by (most cited first, 30 of 46)

- [Automata-based constraints for language model decoding](https://mlsanity.com/paper/2407.08103/) (2024)
- [PentestAgent: Incorporating LLM Agents to Automated Penetration Testing](https://mlsanity.com/paper/2411.05185/) (2024)
- [Vibe Coding vs. Agentic Coding: Fundamentals and Practical Implications of Agentic AI](https://mlsanity.com/paper/2505.19443/) (2025)
- [VeriPlan: Integrating Formal Verification and LLMs into End-User Planning](https://mlsanity.com/paper/2502.17898/) (2025)
- [Towards Visual Text Grounding of Multimodal Large Language Model](https://mlsanity.com/paper/2504.04974/) (2025)
- [SLOT: Structuring the Output of Large Language Models](https://mlsanity.com/paper/2505.04016/) (2025)
- [RuleR: Improving LLM Controllability by Rule-based Data Recycling](https://mlsanity.com/paper/2406.15938/) (2024)
- [TabGen-ICL: Residual-Aware In-Context Example Selection for Tabular Data Generation](https://mlsanity.com/paper/2502.16414/) (2025)
- [CallNavi, A Challenge and Empirical Study on LLM Function Calling and Routing](https://mlsanity.com/paper/2501.05255/) (2025)
- [Breaking the Chain: A Causal Analysis of LLM Faithfulness to Intermediate Structures](https://mlsanity.com/paper/2603.16475/) (2026)
- [TableTalk: Scaffolding Spreadsheet Development with a Language Agent](https://mlsanity.com/paper/2502.09787/) (2025)
- [Generating Structured Plan Representation of Procedures with LLMs](https://mlsanity.com/paper/2504.00029/) (2025)
- [Quantifying the Impact of Structured Output Format on Large Language Models through Causal Inference](https://mlsanity.com/paper/2509.21791/) (2025)
- [Retrieval-Augmented Generation of Ontologies from Relational Databases](https://mlsanity.com/paper/2506.01232/) (2025)
- [EpidemIQs: Prompt-to-Paper LLM Agents for Epidemic Modeling and Analysis](https://mlsanity.com/paper/2510.00024/) (2025)
- [OntoLogX: Ontology-Guided Knowledge Graph Extraction from Cybersecurity Logs with Large Language Models](https://mlsanity.com/paper/2510.01409/) (2025)
- [CogErgLLM: Exploring Large Language Model Systems Design Perspective Using Cognitive Ergonomics](https://mlsanity.com/paper/2407.02885/) (2024)
- [Cancer Type, Stage and Prognosis Assessment from Pathology Reports using LLMs](https://mlsanity.com/paper/2503.01194/) (2025)
- [Large-Scale Constraint Generation -- Can LLMs Parse Hundreds of Constraints?](https://mlsanity.com/paper/2509.24090/) (2025)
- [A Call to Action for a Secure-by-Design Generative AI Paradigm](https://mlsanity.com/paper/2510.00451/) (2025)
- [Constraint Tax in Open-Weight LLMs: An Empirical Study of Tool Calling Suppression Under Structured Output Constraints](https://mlsanity.com/paper/2606.25605/) (2026)
- [ExpressivityBench: Can LLMs Communicate Implicitly?](https://mlsanity.com/paper/2411.08010/) (2024)
- [ACAI for SBOs: AI Co-creation for Advertising and Inspiration for Small Business Owners](https://mlsanity.com/paper/2503.06729/) (2025)
- [Understanding User Experiences of Computer Use Agents: Design Space and Opportunities for Building Agent UX Prototypes](https://mlsanity.com/paper/2510.04452/) (2025)
- [The use of GPT-4o and Other Large Language Models for the Improvement and Design of Self-Assessment Scales for Measurement of Interpersonal Communication Skills](https://mlsanity.com/paper/2409.14050/) (2024)
- [Gensors: Authoring Personalized Visual Sensors with Multimodal Foundation Models and Reasoning](https://mlsanity.com/paper/2501.15727/) (2025)
- [Detecting and Mitigating DDoS Attacks with AI: A Survey](https://mlsanity.com/paper/2503.17867/) (2025)
- [Understanding Prompt Programming Tasks and Questions](https://mlsanity.com/paper/2507.17264/) (2025)
- [You May Speak Freely: Improving the Fine-Grained Visual Recognition Capabilities of Multimodal Large Language Models with Answer Extraction](https://mlsanity.com/paper/2510.14885/) (2025)
- [Using Small Language Models to Reverse-Engineer Machine Learning Pipelines Structures](https://mlsanity.com/paper/2601.03988/) (2026)

## Abstract (arXiv)

Large language models can produce creative and diverse responses. However, to integrate them into current developer workflows, it is essential to constrain their outputs to follow specific formats or standards. In this work, we surveyed 51 experienced industry professionals to understand the range of scenarios and motivations driving the need for output constraints from a user-centered perspective. We identified 134 concrete use cases for constraints at two levels: low-level, which ensures the output adhere to a structured format and an appropriate length, and high-level, which requires the output to follow semantic and stylistic guidelines without hallucination. Critically, applying output constraints could not only streamline the currently repetitive process of developing, testing, and integrating LLM prompts for developers, but also enhance the user experience of LLM-powered features and applications. We conclude with a discussion on user preferences and needs towards articulating intended constraints for LLMs, alongside an initial design for a constraint prototyping tool.