# Small Language Models for Agentic Systems: A Survey of Architectures, Capabilities, and Deployment Trade offs

arXiv: https://arxiv.org/abs/2510.03847 · HTML: https://mlsanity.com/paper/2510.03847/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Placed in General LLM agent frameworks and benchmarks, one of 16 directions in LLM agents and tool use.
- Builds on 3 of its 21 references that are in the map.
- First posted on arXiv 4 Oct 2025.

## Placed in

- LLM agents and tool use / [General LLM agent frameworks and benchmarks](https://mlsanity.com/agents/general-llm-agent-frameworks-and-benchmarks/)

## Builds on (most cited first, 3 of 3)

- [Toolformer: Language Models Can Teach Themselves to Use Tools](https://mlsanity.com/paper/2302.04761/) (2023)
- [QLoRA: Efficient Finetuning of Quantized LLMs](https://mlsanity.com/paper/2305.14314/) (2023)
- [API-Bank: A Comprehensive Benchmark for Tool-Augmented LLMs](https://mlsanity.com/paper/2304.08244/) (2023)

## Built on by (most cited first, 11 of 11)

- [Youtu-LLM: Unlocking the Native Agentic Potential for Lightweight Large Language Models](https://mlsanity.com/paper/2512.24618/) (2025)
- [Robust Tool Use via Fission-GRPO: Learning to Recover from Execution Errors](https://mlsanity.com/paper/2601.15625/) (2026)
- [Think Big, Search Small: Where Capacity Matters in Hierarchical Search Agents?](https://mlsanity.com/paper/2607.07548/) (2026)
- [The "I Don't Know" Filter: Enhancing Agentic Reliability in Function Calling](https://mlsanity.com/paper/2607.04034/) (2026)
- [AgentRouter: Heterogeneous Model Routing for Cost-Optimal Multi-Step Agentic Workflows](https://mlsanity.com/paper/2609.22951/) (2026)
- [Making Prospective Memory SLM-Shaped: Typed Intention Stores for Small-Model Agents](https://mlsanity.com/paper/2609.01272/) (2026)
- [A self-evolving multi-role collaborative framework with fine-grained difficulty guidance for innovative mathematical problem generation](https://mlsanity.com/paper/2601.11792/) (2026)
- [Task-Centric Acceleration of Small-Language Models](https://mlsanity.com/paper/2602.24174/) (2026)
- [Small Reasoning Models are Instruction Followers in Function Calling](https://mlsanity.com/paper/2608.22472/) (2026)
- [Uncertainty Quantification for LLM Agents: A Taxonomy, an Evaluation Protocol, and an Empirical Study](https://mlsanity.com/paper/2609.07395/) (2026)
- [PTC-Decoder: Towards Intelligent SLMs on Offline Resource-Constrained Edge Devices](https://mlsanity.com/paper/2609.30836/) (2026)

## Abstract (arXiv)

Small language models (SLMs; 1-12B params, sometimes up to 20B) are sufficient and often superior for agentic workloads where the objective is schema- and API-constrained accuracy rather than open-ended generation. We synthesize recent evidence across open and proprietary SLMs (Phi-4-Mini, Qwen-2.5-7B, Gemma-2-9B, Llama-3.2-1B/3B, Ministral-3B/8B, Apple on-device 3B, DeepSeek-R1-Distill) and connect it to modern evaluations (BFCL v3/v4, StableToolBench) and serving stacks (vLLM, SGLang, TensorRT-LLM) paired with guided decoding libraries (XGrammar, Outlines). We formalize SLM-default, LLM-fallback systems with uncertainty-aware routing and verifier cascades, and propose engineering metrics that reflect real production goals: cost per successful task (CPS), schema validity rate, executable call rate, p50/p95 latency, and energy per request. Guided decoding, strict JSON Schema outputs, and validator-first tool execution close much of the capability gap with larger models and often let SLMs match or surpass LLMs on tool use, function calling, and RAG at 10x-100x lower token cost with materially better latency and energy. We provide design patterns for agent stacks that prioritize SLMs: schema-first prompting, type-safe function registries, confidence scoring with verifier rollups, and lightweight adaptation via LoRA/QLoRA. We also delineate limits where fallback remains valuable (open-domain reasoning and some long-horizon planning). The result is a practical blueprint for building fast, inexpensive, and reliable agents that default to SLMs while preserving headroom with targeted LLM assistance. Keywords: small language models, agents, function calling, structured outputs, JSON Schema, guided decoding, LoRA/QLoRA, routing, energy efficiency, edge inference