# AgentFlux: Decoupled Fine-Tuning & Inference for On-Device Agentic Systems

arXiv: https://arxiv.org/abs/2510.00229 · HTML: https://mlsanity.com/paper/2510.00229/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Placed in General LLM agent frameworks and benchmarks, one of 16 directions in LLM agents and tool use.
- Builds on 19 of its 27 references that are in the map.
- First posted on arXiv 30 Sep 2025.

## Placed in

- LLM agents and tool use / [General LLM agent frameworks and benchmarks](https://mlsanity.com/agents/general-llm-agent-frameworks-and-benchmarks/)

## Builds on (most cited first, 19 of 19)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://mlsanity.com/paper/2309.06180/) (2023)
- [Qwen2.5 Technical Report](https://mlsanity.com/paper/2412.15115/) (2024)
- [Lost in the Middle: How Language Models Use Long Contexts](https://mlsanity.com/paper/2307.03172/) (2023)
- [GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers](https://mlsanity.com/paper/2210.17323/) (2022)
- [Small Language Models are the Future of Agentic AI](https://mlsanity.com/paper/2506.02153/) (2025)
- [Model Context Protocol (MCP): Landscape, Security Threats, and Future Research Directions](https://mlsanity.com/paper/2503.23278/) (2025)
- [ToolACE: Winning the Points of LLM Function Calling](https://mlsanity.com/paper/2409.00920/) (2024)
- [xLAM: A Family of Large Action Models to Empower AI Agent Systems](https://mlsanity.com/paper/2409.03215/) (2024)
- [MCP-Bench: Benchmarking Tool-Using LLM Agents with Complex Real-World Tasks via MCP Servers](https://mlsanity.com/paper/2508.20453/) (2025)
- [Model Context Protocol (MCP) at First Glance: Studying the Security and Maintainability of MCP Servers](https://mlsanity.com/paper/2506.13538/) (2025)
- [Efficient Inference for Large Reasoning Models: A Survey](https://mlsanity.com/paper/2503.23077/) (2025)
- [TinyAgent: Function Calling at the Edge](https://mlsanity.com/paper/2409.00608/) (2024)
- [Hammer: Robust Function-Calling for On-Device Language Models via Function Masking](https://mlsanity.com/paper/2410.04587/) (2024)
- [The NLP Task Effectiveness of Long-Range Transformers](https://mlsanity.com/paper/2202.07856/) (2022)
- [MCPXKIT: The Unified Toolkit for Analyzing Model Context Protocol Security](https://mlsanity.com/paper/2508.12538/) (2025)
- [ConsumerBench: Benchmarking Generative AI Applications on End-User Devices](https://mlsanity.com/paper/2506.17538/) (2025)

## Built on by (most cited first, 3 of 3)

- [ForkKV: Scaling Multi-LoRA Agent Serving via Copy-on-Write Disaggregated KV Cache](https://mlsanity.com/paper/2604.06370/) (2026)
- [What Stops a Small Language Model From Driving a Database Agent](https://mlsanity.com/paper/2609.21341/) (2026)
- [Should SLMs be in the Driver's Seat? A Study of Local Models as Interpreters in Cloud-Written Agent Programs](https://mlsanity.com/paper/2610.14209/) (2026)

## Abstract (arXiv)

The deployment of Large Language Models (LLMs) as agentic orchestrators has revolutionized task automation, but the need for privacy-preserving, cost-effective solutions demands on-device inference capabilities. However, local LLMs consistently underperform compared to frontier models in tool calling scenarios, struggling with both tool selection from large tool sets and accurate argument generation for complex parameter structures. We introduce a methodology that disaggregates a tool-calling task into two distinct subtasks: tool selection and argument generation. We propose "decoupled fine-tuning", a novel post-training approach that employs LoRA fine-tuning to create dedicated LoRA adapters for tool selection and tool-specific argument generation using separate loss masking for each of the subtasks. Furthermore, we present AgentFlux, an inference framework that leverages the LoRA adapters created using decoupled fine-tuning to perform efficient agent orchestration with the help of local models on end-user devices. AgentFlux decomposes the tool-call generation step into tool selection and argument generation, and dynamically loads the corresponding LoRA adapters to generate tool calls. Additionally, AgentFlux implements hierarchical orchestration to restrict the number of tools required for tool selection. Our experiments on the MCP-Bench benchmark demonstrate that the Qwen-2.5-7B model trained using decoupled fine-tuning improves the tool calling accuracy of the base model by 46%, and outperforms other local reasoning, non-reasoning and fine-tuned models of similar size in all cases, and models that are 2x larger, in most cases.