mlsanityData to 14 Oct 2026

Routine: A Structural Planning Framework for LLM Agent System in Enterprise

arXiv 2507.14447 ↗First posted 19 Jul 2025cs.AI cs.CLcited by 9

On the map: LLM agents and tool use → General LLM agent frameworks and benchmarks

Abstract (arXiv)

The deployment of agent systems in an enterprise environment is often hindered by several challenges: common models lack domain-specific process knowledge, leading to disorganized plans, missing key tools, and poor execution stability. To address this, this paper introduces Routine, a multi-step agent planning framework designed with a clear structure, explicit instructions, and seamless parameter passing to guide the agent's execution module in performing multi-step tool-calling tasks with high stability. In evaluations conducted within a real-world enterprise scenario, Routine significantly increases the execution accuracy in model tool calls, increasing the performance of GPT-4o from 41.1% to 96.3%, and Qwen3-14B from 32.6% to 83.3%. We further constructed a Routine-following training dataset and fine-tuned Qwen3-14B, resulting in an accuracy increase to 88.2% on scenario-specific evaluations, indicating improved adherence to execution plans. In addition, we employed Routine-based distillation to create a scenario-specific, multi-step tool-calling dataset. Fine-tuning on this distilled dataset raised the model's accuracy to 95.5%, approaching GPT-4o's performance. These results highlight Routine's effectiveness in distilling domain-specific tool-usage patterns and enhancing model adaptability to new scenarios. Our experimental results demonstrate that Routine provides a practical and accessible approach to building stable agent workflows, accelerating the deployment and adoption of agent systems in enterprise environments, and advancing the technical vision of AI for Process.

Key facts. Cited by 9 papers in the map (data through 14 Oct 2026). Placed in General LLM agent frameworks and benchmarks, one of 16 directions in LLM agents and tool use. Builds on 37 of its 67 references that are in the map. First posted on arXiv 19 Jul 2025.

Builds on

37 of 67 references.

Closest ancestors · 9

Same direction, newest first

  1. 2025The Cost of Dynamic Reasoning: Demystifying AI Agents and Test-Time Scaling from an AI Infrastructure Perspective4 Jun
  2. 2025Runaway is Ashamed, But Helpful: On the Early-Exit Behavior of Large Language Model-based Agents in Embodied Environments23 May
  3. 2025GuideBench: Benchmarking Domain-Oriented Guideline Following for LLM Agents16 May
  4. 2025From Mind to Machine: The Rise of Manus AI as a Fully Autonomous Digital Agent4 May
  5. 2025Orchestrating Agents and Data for Enterprise: A Blueprint Architecture for Compound AI10 Apr
  6. 2025Evaluating LLM-based Agents for Multi-Turn Conversations: A Survey28 Mar
  7. 2025Plan-and-Act: Improving Planning of Agents for Long-Horizon Tasks12 Mar
  8. 2023A Survey on Large Language Model based Autonomous Agents22 Aug
  9. 2023Multi-Agent Collaboration: Harnessing the Power of Intelligent LLM Agents5 Jun

Other references · 28

  1. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  2. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  3. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
  4. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,724
  5. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,598
  6. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,470
  7. 2023Gorilla: Large Language Model Connected with Massive APIscited by 1,332
  8. 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,301
  9. 2023Instruction-Following Evaluation for Large Language Modelscited by 1,228
  10. 2023ToolAlpaca: Generalized Tool Learning for Language Models with 3000 Simulated Casescited by 393
  11. 2024LongRoPE: Extending LLM Context Window Beyond 2 Million Tokenscited by 313
  12. 2024Tool Learning with Large Language Models: A Surveycited by 223
16 more
  1. 2024ToolACE: Winning the Points of LLM Function Callingcited by 216
  2. 2024APIGen: Automated Pipeline for Generating Verifiable and Diverse Function-Calling Datasetscited by 198
  3. 2024Benchmarking Complex Instruction-Following with Multiple Constraints Compositioncited by 152
  4. 2025Does Math Reasoning Improve General LLM Capabilities? Understanding Transferability of LLM Reasoningcited by 106
  5. 2025MCP-RADAR: A Multi-Dimensional Benchmark for Evaluating Tool Use Capabilities in Large Language Modelscited by 48
  6. 2024Reducing Tool Hallucination via Reliability Alignmentcited by 39
  7. 2024Tool-Planner: Task Planning with Clusters across Multiple Toolscited by 33
  8. 2024IOPO: Empowering LLMs with Complex Instruction Following via Input-Output Preference Optimizationcited by 31
  9. 2023MANUS: Markerless Grasp Capture using Articulated 3D Gaussianscited by 26
  10. 2025LongFuncEval: Measuring the effectiveness of long context models for function callingcited by 18
  11. 2025A Survey on Progress in LLM Alignment from the Perspective of Reward Designcited by 15
  12. 2025Synthetic Data RL: Task Definition Is All You Needcited by 15
  13. 2024IDA: Breaking Barriers in No-code UI Automation Through Large Language Models and Human-Centric Designcited by 5
  14. 2025Task Memory Engine: Spatial Memory for Robust Multi-Step LLM Agentscited by 5
  15. 2025Evaluating Large Language Models in Vulnerability Detection Under Variable Context Windowscited by 2
  16. 2025Extracting Knowledge Graphs from User Stories using LangChaincited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 8.

9 papers.

  1. 2026BackdoorAgent: A Unified Framework for Backdoor Attacks on LLM-based Agentscited by 16
  2. 2026Your Agents Are Aging Too: Agent Lifespan Engineering for Deployed Systemscited by 9
  3. 2026Table-as-Search: Formulate Long-Horizon Agentic Information Seeking as Table Completioncited by 5
  4. 2026From Agent Loops to Structured Graphs:A Scheduler-Theoretic Framework for LLM Agent Executioncited by 4
  5. 2025Empowering Real-World: A Survey on the Technology, Practice, and Evaluation of LLM-driven Industry Agentscited by 2
  6. 2026Tools as Continuous Flow for Evolving Agentic Reasoningcited by 2
  7. 2026Bridging Policy and Real-World Dynamics: LLM-Augmented Rebalancing for Shared Micromobility Systemscited by 0
  8. 2026Small Reasoning Models are Instruction Followers in Function Callingcited by 0
  9. 2026Mem++: Non-Destructive Memory for Long-Term Organizational LLM Agentscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Architectural Design, Not Only Model Intelligence, Governs Multi-Agent LLM Performance
  2. 2026Specialize Roles, Mix Deployments: Pushing the Cost-Accuracy Frontier of LLM Agent Teams
  3. 2026The Evolution of Tool Use in LLM Agents: From Single-Tool Call to Multi-Tool Orchestration
  4. 2024Adaptable and Precise: Enterprise-Scenario LLM Function-Calling Capability Training Pipeline
  5. 2026AgentFloor: How Far Up the tool use Ladder Can Small Open-Weight Models Go?
  6. 2026Agent Planning Benchmark: A Diagnostic Framework for Planning Capabilities in LLM Agents
  7. 2025Measuring Agents in Production
  8. 2024FlowBench: Revisiting and Benchmarking Workflow-Guided Planning for LLM-based Agents