Declarative by Design, Assistable Only by Convention: Benchmarking Multi-Agent Frameworks for AI-Assistability

arXiv 2602.11198 ↗First posted 3 Feb 2026cs.CL cs.AIcited by 0

On the map: LLM agents and tool use → Software engineering coding agents

Abstract (arXiv)

Multi-agent frameworks (MAFs) promise to simplify LLM-driven software development, yet no principled metric captures how well AI coding assistants can generate correct, framework-specific code. We introduce textitAI-assistability (AI), a composite metric that quantifies a framework's amenability to AI-assisted development by combining structural alignment (barσ) with functional correctness (pass@1). To evaluate this metric in a controlled setting, we design DDL2PropBank, a novel benchmark task that maps relational database schemas to PropBank semantic rolesets, and implement identical agent logic across ten frameworks using the Agent-as-a-Tool pattern. Our results challenge the intuition that declarative framework design guarantees AI-assistability: Agno, with a single canonical pattern and convention-aligned API, achieves the highest AI score (0.55), while DSPy -- the most declarative framework by design -- scores lowest (0.07), as its novel abstractions are insufficiently represented in AI training data. We find that convention alignment, not declarative design alone, is the primary driver of AI-assistability (r = 0.576 between barσ and pass@1). All artifacts -- DDL2PropBank, PropBank MCP server, and all implementations -- are available at https://github.com/ahmeshaf/ddl2propbank

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Software engineering coding agents, one of 16 directions in LLM agents and tool use. Builds on 21 of its 53 references that are in the map. First posted on arXiv 3 Feb 2026.

Builds on

21 of 53 references.

Closest ancestors · 1

Same direction, newest first

2025An Empirical Study of Agent Developer Practices in AI Agent Frameworks1 Dec

Other references · 20

2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
2021Evaluating Large Language Models Trained on Codecited by 8,053
2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,072
2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,205
2023DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelinescited by 803
2023Generative agent-based modeling with actions grounded in physical, social, or digital space using Concordiacited by 119
2023CoAnnotating: Uncertainty-Guided Work Allocation between Human and Large Language Models for Data Annotationcited by 93
2024RelBench: A Benchmark for Deep Learning on Relational Databasescited by 90
2024CheckEval: A reliable LLM-as-a-Judge framework for evaluating text generation using checklistscited by 87
2024A Review of Prominent Paradigms for LLM-Based Agents: Tool Use (Including RAG), Planning, and Feedback Learningcited by 74
2025RelGNN: Composite Message Passing for Relational Deep Learningcited by 45
2024Exploration of LLM Multi-Agent Application Implementation Based on LangGraph+CrewAIcited by 35

8 more

2024Simulating Human-like Daily Activities with Desire-driven Autonomycited by 30
2024Synthesizing Post-Training Data for LLMs through Multi-Agent Simulationcited by 22
2025Agentic AI Frameworks: Architectures, Protocols, and Design Challengescited by 16
2025AgentScope 1.0: A Developer-Centric Framework for Building Agentic Applicationscited by 16
2024Why LLMs Are Bad at Synthetic Table Generation (and what to do about it)cited by 9
2025REDELEX: A Framework for Relational Deep Learning Explorationcited by 8
2024Linear Cross-document Event Coreference Resolution with X-AMRcited by 6
2024Schema-Guided Culture-Aware Complex Event Simulation with Multi-Agent Role-Playcited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way