# Division-of-Thoughts: Harnessing Hybrid Language Model Synergy for Efficient On-Device Agents

arXiv: https://arxiv.org/abs/2502.04392 · HTML: https://mlsanity.com/paper/2502.04392/

## Key facts

- Cited by 25 papers in the map (data through 14 Oct 2026).
- Placed in Reasoning for agents, planning and code, one of 17 directions in LLM reasoning.
- Builds on 27 of its 45 references that are in the map.
- First posted on arXiv 6 Feb 2025.

## Placed in

- LLM reasoning / [Reasoning for agents, planning and code](https://mlsanity.com/lr/reasoning-for-agents-planning-and-code/)
- LLM agents and tool use / [General LLM agent frameworks and benchmarks](https://mlsanity.com/agents/general-llm-agent-frameworks-and-benchmarks/)

## Builds on (most cited first, 27 of 27)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Measuring Mathematical Problem Solving With the MATH Dataset](https://mlsanity.com/paper/2103.03874/) (2021)
- [Tree of Thoughts: Deliberate Problem Solving with Large Language Models](https://mlsanity.com/paper/2305.10601/) (2023)
- [Emergent Abilities of Large Language Models](https://mlsanity.com/paper/2206.07682/) (2022)
- [Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone](https://mlsanity.com/paper/2404.14219/) (2024)
- [CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledge](https://mlsanity.com/paper/1811.00937/) (2018)
- [Least-to-Most Prompting Enables Complex Reasoning in Large Language Models](https://mlsanity.com/paper/2205.10625/) (2022)
- [DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphs](https://mlsanity.com/paper/1903.00161/) (2019)
- [FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance](https://mlsanity.com/paper/2305.05176/) (2023)
- [A Survey on In-context Learning](https://mlsanity.com/paper/2301.00234/) (2022)
- [Decomposed Prompting: A Modular Approach for Solving Complex Tasks](https://mlsanity.com/paper/2210.02406/) (2022)
- [Towards Large Reasoning Models: A Survey of Reinforced Reasoning with Large Language Models](https://mlsanity.com/paper/2501.09686/) (2025)
- [Personal LLM Agents: Insights and Survey about the Capability, Efficiency and Security](https://mlsanity.com/paper/2401.05459/) (2024)
- [AgentSquare: Automatic LLM Agent Search in Modular Design Space](https://mlsanity.com/paper/2410.06153/) (2024)
- [Language Model Cascades: Token-level uncertainty and beyond](https://mlsanity.com/paper/2404.10136/) (2024)
- [Small Language Models: Survey, Measurements, and Insights](https://mlsanity.com/paper/2409.15790/) (2024)
- [On-Device Language Models: A Comprehensive Review](https://mlsanity.com/paper/2409.00088/) (2024)
- [A Survey of Resource-efficient LLM and Multimodal Foundation Models](https://mlsanity.com/paper/2401.08092/) (2024)
- [Scaling Sentence Embeddings with Large Language Models](https://mlsanity.com/paper/2307.16645/) (2023)
- [CHAMP: A Competition-level Dataset for Fine-Grained Analyses of LLMs' Mathematical Reasoning Capabilities](https://mlsanity.com/paper/2401.06961/) (2024)
- [Programming Puzzles](https://mlsanity.com/paper/2106.05784/) (2021)
- [Apple Intelligence Foundation Language Models](https://mlsanity.com/paper/2407.21075/) (2024)
- [Chain-of-Planned-Behaviour Workflow Elicits Few-Shot Mobility Generation in LLMs](https://mlsanity.com/paper/2402.09836/) (2024)
- [Synergy-of-Thoughts: Eliciting Efficient Reasoning in Hybrid Language Models](https://mlsanity.com/paper/2402.02563/) (2024)
- [Octo-planner: On-device Language Model for Planner-Action Agents](https://mlsanity.com/paper/2406.18082/) (2024)
- [LIMP: Large Language Model Enhanced Intent-aware Mobility Prediction](https://mlsanity.com/paper/2408.12832/) (2024)
- [Harnessing the Synergy between LLM Agents and Knowledge Graphs for Urban Socioeconomic Prediction](https://mlsanity.com/paper/2411.00028/) (2024)

## Built on by (most cited first, 25 of 25)

- [A Survey of WebAgents: Towards Next-Generation AI Agents for Web Automation with Large Foundation Models](https://mlsanity.com/paper/2503.23350/) (2025)
- [Get Experience from Practice: LLM Agents with Record & Replay](https://mlsanity.com/paper/2505.17716/) (2025)
- [Route to Reason: Adaptive Routing for LLM and Reasoning Strategy Selection](https://mlsanity.com/paper/2505.19435/) (2025)
- [FingerTip 20K: A Benchmark for Proactive and Personalized Mobile LLM Agents](https://mlsanity.com/paper/2507.21071/) (2025)
- [AgentSwift: Efficient LLM Agent Design via Value-guided Hierarchical Search](https://mlsanity.com/paper/2506.06017/) (2025)
- [A Survey on Collaborative Mechanisms Between Large and Small Language Models](https://mlsanity.com/paper/2505.07460/) (2025)
- [Fate: Fast Edge Inference of Mixture-of-Experts Models via Cross-Layer Gate](https://mlsanity.com/paper/2502.12224/) (2025)
- [Route-and-Reason: Scaling Large Language Model Reasoning with Reinforced Model Router](https://mlsanity.com/paper/2506.05901/) (2025)
- [AgentSociety Challenge: Designing LLM Agents for User Modeling and Recommendation on Web Platforms](https://mlsanity.com/paper/2502.18754/) (2025)
- [Agent.xpu: Efficient Scheduling of Agentic LLM Workloads on Heterogeneous SoC](https://mlsanity.com/paper/2506.24045/) (2025)
- [Automating High Energy Physics Data Analysis with LLM-Powered Agents](https://mlsanity.com/paper/2512.07785/) (2025)
- [A Survey on Collaborating Small and Large Language Models for Performance, Cost-effectiveness, Cloud-edge Privacy, and Trustworthiness](https://mlsanity.com/paper/2510.13890/) (2025)
- [FlowSteer: Prompt-Only Workflow Steering Exposes Planning-Time Vulnerabilities in Multi-Agent LLM Systems](https://mlsanity.com/paper/2605.11514/) (2026)
- [Multi-level Value Alignment in Agentic AI Systems: Survey and Perspectives](https://mlsanity.com/paper/2506.09656/) (2025)
- [Bridging On-Device and Cloud LLMs for Collaborative Reasoning: A Unified Methodology for Local Routing and Post-Training](https://mlsanity.com/paper/2509.24050/) (2025)
- [PEAR: Planner-Executor Agent Robustness Benchmark](https://mlsanity.com/paper/2510.07505/) (2025)
- [Token Signature: Predicting Chain-of-Thought Gains with Token Decoding Feature in Large Language Models](https://mlsanity.com/paper/2506.06008/) (2025)
- [Towards Pervasive Distributed Agentic Generative AI -- A State of The Art](https://mlsanity.com/paper/2506.13324/) (2025)
- [Small-Large Collaboration: Training-efficient Concept Personalization for Large VLM using a Meta Personalized Small VLM](https://mlsanity.com/paper/2508.07260/) (2025)
- [Federated Learning-Enabled Hybrid Language Models for Communication-Efficient Token Transmission](https://mlsanity.com/paper/2507.00082/) (2025)
- [When Do LLMs Reason? A Dynamical Systems View via Entropy Phase Transitions](https://mlsanity.com/paper/2605.22873/) (2026)
- [Cloud-Device Collaborative Agents for Sequential Recommendation](https://mlsanity.com/paper/2509.01551/) (2025)
- [Uncertainty-Aware Collaborative System of Large and Small Models for Multimodal Sentiment Analysis](https://mlsanity.com/paper/2509.04459/) (2025)
- [IntPro: A Proxy Agent for Context-Aware Intent Understanding via Retrieval-conditioned Inference](https://mlsanity.com/paper/2603.03325/) (2026)
- [Empowering Edge Agents: A Systematic Analysis of Edge-Cloud Collaboration in LLM Agent Execution](https://mlsanity.com/paper/2610.17816/) (2026)

## Abstract (arXiv)

The rapid expansion of web content has made on-device AI assistants indispensable for helping users manage the increasing complexity of online tasks. The emergent reasoning ability in large language models offer a promising path for next-generation on-device AI agents. However, deploying full-scale Large Language Models (LLMs) on resource-limited local devices is challenging. In this paper, we propose Division-of-Thoughts (DoT), a collaborative reasoning framework leveraging the synergy between locally deployed Smaller-scale Language Models (SLMs) and cloud-based LLMs. DoT leverages a Task Decomposer to elicit the inherent planning abilities in language models to decompose user queries into smaller sub-tasks, which allows hybrid language models to fully exploit their respective strengths. Besides, DoT employs a Task Scheduler to analyze the pair-wise dependency of sub-tasks and create a dependency graph, facilitating parallel reasoning of sub-tasks and the identification of key steps. To allocate the appropriate model based on the difficulty of sub-tasks, DoT leverages a Plug-and-Play Adapter, which is an additional task head attached to the SLM that does not alter the SLM's parameters. To boost adapter's task allocation capability, we propose a self-reinforced training method that relies solely on task execution feedback. Extensive experiments on various benchmarks demonstrate that our DoT significantly reduces LLM costs while maintaining competitive reasoning accuracy. Specifically, DoT reduces the average reasoning time and API costs by 66.12% and 83.57%, while achieving comparable reasoning accuracy with the best baseline methods.