mlsanityData to 7 Oct 2026

Prompting Is Programming: A Query Language for Large Language Models

arXiv 2212.06094 ↗First posted 12 Dec 2022cs.CL cs.AIcited by 110

Not on the map. Why?

Abstract (arXiv)

Large language models have demonstrated outstanding performance on a wide range of tasks such as question answering and code generation. On a high level, given an input, a language model can be used to automatically complete the sequence in a statistically-likely way. Based on this, users prompt these models with language instructions or examples, to implement a variety of downstream tasks. Advanced prompting methods can even imply interaction between the language model, a user, and external tools such as calculators. However, to obtain state-of-the-art performance or adapt language models for specific tasks, complex task- and model-specific programs have to be implemented, which may still require ad-hoc interaction. Based on this, we present the novel idea of Language Model Programming (LMP). LMP generalizes language model prompting from pure text prompts to an intuitive combination of text prompting and scripting. Additionally, LMP allows constraints to be specified over the language model output. This enables easy adaption to many tasks while abstracting language model internals and providing high-level semantics. To enable LMP, we implement LMQL(short for Language Model Query Language), which leverages the constraints and control flow from an LMP prompt to generate an efficient inference procedure that minimizes the number of expensive calls to the underlying language model. We show that LMQL can capture a wide range of state-of-the-art prompting methods in an intuitive way, especially facilitating interactive flows that are challenging to implement with existing high-level APIs. Our evaluation shows that we retain or increase the accuracy on several downstream tasks, while also significantly reducing the required amount of computation or cost in the case of pay-to-use APIs (26-85% cost savings).

Key facts. Cited by 110 papers in the map (data through 7 Oct 2026). Builds on 24 of its 43 references that are in the map. First posted on arXiv 12 Dec 2022.

Builds on

24 of 43 references.

References · 24

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  5. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  6. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  7. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  8. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,686
  9. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  10. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  11. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,970
  12. 2022Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Taskscited by 1,057
12 more
  1. 2022Large Language Models Are Human-Level Prompt Engineerscited by 1,020
  2. 2022PAL: Program-aided Language Modelscited by 635
  3. 2021Prompt Programming for Large Language Models: Beyond the Few-Shot Paradigmcited by 599
  4. 2021PICARD: Parsing Incrementally for Constrained Auto-Regressive Decoding from Language Modelscited by 402
  5. 2022PromptSource: An Integrated Development Environment and Repository for Natural Language Promptscited by 288
  6. 2021Constrained Language Models Yield Few-Shot Semantic Parserscited by 162
  7. 2022Synchromesh: Reliable code generation from pre-trained language modelscited by 156
  8. 2021OpenPrompt: An Open-source Framework for Prompt-learningcited by 148
  9. 2022PromptChainer: Chaining Large Language Model Prompts through Visual Programmingcited by 134
  10. 2022Language Model Cascadescited by 104
  11. 2019Giving BERT a Calculator: Finding Operations and Arguments with Reading Comprehensioncited by 51
  12. 2022ThinkSum: Probabilistic reasoning over sets using large language modelscited by 23

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 2; 2023: 23; 2024: 18; 2025: 24; 2026: 43.

110 papers.

Sort:

By area:

  1. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,107
  2. 2023DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelinescited by 803
  3. 2023Large Language Models for Software Engineering: A Systematic Literature Reviewcited by 389
  4. 2023Prompt Cache: Modular Attention Reuse for Low-Latency Inferencecited by 255
  5. 2022Reasoning with Language Model Prompting: A Surveycited by 251
  6. 2026Meta-Harness: End-to-End Optimization of Model Harnessescited by 248
  7. 2023ART: Automatic multi-step reasoning and tool-use for large language modelscited by 193
  8. 2023Efficient Guided Generation for Large Language Modelscited by 192
  9. 2023Amortizing intractable inference in large language modelscited by 94
  10. 2023ChainForge: A Visual Toolkit for Prompt Engineering and LLM Hypothesis Testingcited by 88
  11. 2023Sequential Monte Carlo Steering of Large Language Models using Probabilistic Programscited by 79
  12. 2024Safeguarding Large Language Models: A Surveycited by 73
38 more
  1. 2025JSONSchemaBench: A Rigorous Benchmark of Structured Outputs for Language Modelscited by 66
  2. 2022Parsel: Algorithmic Reasoning with Language Models by Composing Decompositionscited by 62
  3. 2024Agents in Software Engineering: Survey, Landscape, and Visioncited by 60
  4. 2026Natural-Language Agent Harnessescited by 54
  5. 2024Tutor CoPilot: A Human-AI Approach for Scaling Real-Time Expertisecited by 46
  6. 2023Neural Graph Reasoning: Complex Logical Query Answering Meets Graph Databasescited by 35
  7. 2023What's Left? Concept Grounding with Logic-Enhanced Foundation Modelscited by 33
  8. 2024SPML: A DSL for Defending Language Models Against Prompt Attackscited by 33
  9. 2025Type-Constrained Code Generation with Language Modelscited by 32
  10. 2024IterGen: Iterative Semantic-aware Structured LLM Generation with Backtrackingcited by 28
  11. 2023DSPy Assertions: Computational Constraints for Self-Refining Language Model Pipelinescited by 27
  12. 2024StructuredRAG: JSON Response Formatting with Large Language Modelscited by 22
  13. 2024UQE: A Query Engine for Unstructured Databasescited by 18
  14. 2024AgentKit: Structured LLM Reasoning with Dynamic Graphscited by 12
  15. 2023Kani: A Lightweight and Highly Hackable Framework for Building Language Model Applicationscited by 11
  16. 2024Natural language is not enough: Benchmarking multi-modal generative AI for Verilog generationcited by 11
  17. 2024ScenicNL: Generating Probabilistic Scenario Programs from Natural Languagecited by 10
  18. 2023Numeracy from Literacy: Data Science as an Emergent Skill from Large Language Modelscited by 8
  19. 2023A Unified Industrial Large Knowledge Model Framework in Industry 4.0 and Smart Manufacturingcited by 8
  20. 2024APPL: A Prompt Programming Language for Harmonious Integration of Programs and Large Language Model Promptscited by 8
  21. 2024A Perspective for Adapting Generalist AI to Specialized Medical AI Applications and Their Challengescited by 8
  22. 2026SEVerA: Verified Synthesis of Self-Evolving Agentscited by 6
  23. 2023AskIt: Unified Programming Interface for Programming with Large Language Modelscited by 5
  24. 2024Understanding the Logic of Direct Preference Alignment through Logiccited by 5
  25. 2025Leveraging Vision-Language Models for Open-Vocabulary Instance Segmentation and Trackingcited by 4
  26. 2025Semantic Probabilistic Control of Language Modelscited by 4
  27. 2026The Constraint Tax: Measuring Validity-Correctness Tradeoffs in Structured Outputs for Small Language Modelscited by 4
  28. 2023Toward Reproducing Network Research Results Using Large Language Modelscited by 3
  29. 2023Automatically Testing Functional Properties of Code Translation Modelscited by 3
  30. 2023Prompt Sketching for Large Language Modelscited by 3
  31. 2024MTP: A Meaning-Typed Language Abstraction for AI-Integrated Programmingcited by 3
  32. 2024AutoFeedback: An LLM-based Framework for Efficient and Accurate API Request Generationcited by 3
  33. 2025Earley-Driven Dynamic Pruning for Efficient Structured Decodingcited by 3
  34. 2025Constrained Decoding for Safe Robot Navigation Foundation Modelscited by 3
  35. 2025Constrained Adaptive Rejection Samplingcited by 3
  36. 2026SkillSmith: Compiling Agent Skills into Boundary-Guided Runtime Interfacescited by 3
  37. 2026From Prompts to Contracts: Harness Engineering for Auditable Enterprise LLM Agentscited by 3
  38. 2026NVIDIA-labs OO Agents: Native Python Object-Oriented Agentscited by 3

50 most cited of 110 All 110 (JSON)