# Specifications: The missing link to making the development of LLM systems an engineering discipline

arXiv: https://arxiv.org/abs/2412.05299 · HTML: https://mlsanity.com/paper/2412.05299/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 25 Nov 2024.

## Built on by (most cited first, 17 of 17)

- [Why Do Multi-Agent LLM Systems Fail?](https://mlsanity.com/paper/2503.13657/) (2025)
- [Model Context Protocol (MCP) Tool Descriptions Are Smelly! Towards Improving AI Agent Efficiency with Augmented MCP Tool Descriptions](https://mlsanity.com/paper/2602.14878/) (2026)
- [VISION: A Modular AI Assistant for Natural Human-Instrument Interaction at Scientific User Facilities](https://mlsanity.com/paper/2412.18161/) (2024)
- [PromptPex: Automatic Test Generation for Language Model Prompts](https://mlsanity.com/paper/2503.05070/) (2025)
- [Position: Scaling LLM Agents Requires Asymptotic Analysis with LLM Primitives](https://mlsanity.com/paper/2502.04358/) (2025)
- [Toward a Safe Internet of Agents](https://mlsanity.com/paper/2512.00520/) (2025)
- [Willful Disobedience: Automatically Detecting Failures in Agentic Traces](https://mlsanity.com/paper/2603.23806/) (2026)
- [Standardizing Intelligence: Aligning Generative AI for Regulatory and Operational Compliance](https://mlsanity.com/paper/2503.04736/) (2025)
- [Chatty-KG: A Multi-Agent AI System for On-Demand Conversational Question Answering over Knowledge Graphs](https://mlsanity.com/paper/2511.20940/) (2025)
- [Querying Databases with Function Calling](https://mlsanity.com/paper/2502.00032/) (2025)
- [SysMoBench: Evaluating AI on Formally Modeling Complex Real-World Systems](https://mlsanity.com/paper/2509.23130/) (2025)
- [Empirical Computation: Prompting versus Programming](https://mlsanity.com/paper/2503.10954/) (2025)
- [Octopus Protocol: One-Shot Hardware Discovery and Control for AI Agents via Infrastructure-as-Prompts](https://mlsanity.com/paper/2605.09055/) (2026)
- [Engineering Robustness into Personal Agents with the AI Workflow Store](https://mlsanity.com/paper/2605.10907/) (2026)
- [The Time is Here for Just-in-Time Systems: Challenges and Opportunities](https://mlsanity.com/paper/2605.24096/) (2026)
- [AssumptionMiner: Extracting, Tracing, and Revising Implicit Assumptions in LLM Code Generation](https://mlsanity.com/paper/2607.22898/) (2026)
- [Reality Is the Final Verifier: On Two Key Gaps in Agentic Software Engineering](https://mlsanity.com/paper/2609.12039/) (2026)

## Abstract (arXiv)

Despite the significant strides made by generative AI in just a few short years, its future progress is constrained by the challenge of building modular and robust systems. This capability has been a cornerstone of past technological revolutions, which relied on combining components to create increasingly sophisticated and reliable systems. Cars, airplanes, computers, and software consist of components-such as engines, wheels, CPUs, and libraries-that can be assembled, debugged, and replaced. A key tool for building such reliable and modular systems is specification: the precise description of the expected behavior, inputs, and outputs of each component. However, the generality of LLMs and the inherent ambiguity of natural language make defining specifications for LLM-based components (e.g., agents) both a challenging and urgent problem. In this paper, we discuss the progress the field has made so far-through advances like structured outputs, process supervision, and test-time compute-and outline several future directions for research to enable the development of modular and reliable LLM-based systems through improved specifications.