# Declarative by Design, Assistable Only by Convention: Benchmarking Multi-Agent Frameworks for AI-Assistability

arXiv: https://arxiv.org/abs/2602.11198 · HTML: https://mlsanity.com/paper/2602.11198/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Software engineering coding agents, one of 16 directions in LLM agents and tool use.
- Builds on 21 of its 53 references that are in the map.
- First posted on arXiv 3 Feb 2026.

## Placed in

- LLM agents and tool use / [Software engineering coding agents](https://mlsanity.com/agents/software-engineering-coding-agents/)

## Builds on (most cited first, 21 of 21)

- [ReAct: Synergizing Reasoning and Acting in Language Models](https://mlsanity.com/paper/2210.03629/) (2022)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [Toolformer: Language Models Can Teach Themselves to Use Tools](https://mlsanity.com/paper/2302.04761/) (2023)
- [G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment](https://mlsanity.com/paper/2303.16634/) (2023)
- [DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelines](https://mlsanity.com/paper/2310.03714/) (2023)
- [Generative agent-based modeling with actions grounded in physical, social, or digital space using Concordia](https://mlsanity.com/paper/2312.03664/) (2023)
- [RelBench: A Benchmark for Deep Learning on Relational Databases](https://mlsanity.com/paper/2407.20060/) (2024)
- [CoAnnotating: Uncertainty-Guided Work Allocation between Human and Large Language Models for Data Annotation](https://mlsanity.com/paper/2310.15638/) (2023)
- [CheckEval: A reliable LLM-as-a-Judge framework for evaluating text generation using checklists](https://mlsanity.com/paper/2403.18771/) (2024)
- [A Review of Prominent Paradigms for LLM-Based Agents: Tool Use (Including RAG), Planning, and Feedback Learning](https://mlsanity.com/paper/2406.05804/) (2024)
- [RelGNN: Composite Message Passing for Relational Deep Learning](https://mlsanity.com/paper/2502.06784/) (2025)
- [Exploration of LLM Multi-Agent Application Implementation Based on LangGraph+CrewAI](https://mlsanity.com/paper/2411.18241/) (2024)
- [Simulating Human-like Daily Activities with Desire-driven Autonomy](https://mlsanity.com/paper/2412.06435/) (2024)
- [Synthesizing Post-Training Data for LLMs through Multi-Agent Simulation](https://mlsanity.com/paper/2410.14251/) (2024)
- [AgentScope 1.0: A Developer-Centric Framework for Building Agentic Applications](https://mlsanity.com/paper/2508.16279/) (2025)
- [Agentic AI Frameworks: Architectures, Protocols, and Design Challenges](https://mlsanity.com/paper/2508.10146/) (2025)
- [An Empirical Study of Agent Developer Practices in AI Agent Frameworks](https://mlsanity.com/paper/2512.01939/) (2025)
- [Why LLMs Are Bad at Synthetic Table Generation (and what to do about it)](https://mlsanity.com/paper/2406.14541/) (2024)
- [REDELEX: A Framework for Relational Deep Learning Exploration](https://mlsanity.com/paper/2506.22199/) (2025)
- [Linear Cross-document Event Coreference Resolution with X-AMR](https://mlsanity.com/paper/2404.08656/) (2024)
- [Schema-Guided Culture-Aware Complex Event Simulation with Multi-Agent Role-Play](https://mlsanity.com/paper/2410.18935/) (2024)

## Abstract (arXiv)

Multi-agent frameworks (MAFs) promise to simplify LLM-driven software development, yet no principled metric captures how well AI coding assistants can generate correct, framework-specific code. We introduce textitAI-assistability (AI), a composite metric that quantifies a framework's amenability to AI-assisted development by combining structural alignment (barσ) with functional correctness (pass@1). To evaluate this metric in a controlled setting, we design DDL2PropBank, a novel benchmark task that maps relational database schemas to PropBank semantic rolesets, and implement identical agent logic across ten frameworks using the Agent-as-a-Tool pattern. Our results challenge the intuition that declarative framework design guarantees AI-assistability: Agno, with a single canonical pattern and convention-aligned API, achieves the highest AI score (0.55), while DSPy -- the most declarative framework by design -- scores lowest (0.07), as its novel abstractions are insufficiently represented in AI training data. We find that convention alignment, not declarative design alone, is the primary driver of AI-assistability (r = 0.576 between barσ and pass@1). All artifacts -- DDL2PropBank, PropBank MCP server, and all implementations -- are available at https://github.com/ahmeshaf/ddl2propbank