# A Survey on Collaborative Mechanisms Between Large and Small Language Models

arXiv: https://arxiv.org/abs/2505.07460 · HTML: https://mlsanity.com/paper/2505.07460/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 69 of its 93 references that are in the map.
- First posted on arXiv 12 May 2025.

## Builds on (most cited first, 30 of 69)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [ALBERT: A Lite BERT for Self-supervised Learning of Language Representations](https://mlsanity.com/paper/1909.11942/) (2019)
- [A Comprehensive Overview of Large Language Models](https://mlsanity.com/paper/2307.06435/) (2023)
- [Hybrid LLM: Cost-Efficient and Quality-Aware Query Routing](https://mlsanity.com/paper/2404.14618/) (2024)
- [A Survey on Knowledge Distillation of Large Language Models](https://mlsanity.com/paper/2402.13116/) (2024)
- [Model Merging in LLMs, MLLMs, and Beyond: Methods, Theories, Applications and Opportunities](https://mlsanity.com/paper/2408.07666/) (2024)
- [Chain of Agents: Large Language Models Collaborating on Long-Context Tasks](https://mlsanity.com/paper/2406.02818/) (2024)
- [VITA-1.5: Towards GPT-4o Level Real-Time Vision and Speech Interaction](https://mlsanity.com/paper/2501.01957/) (2025)
- [Llumnix: Dynamic Scheduling for Large Language Model Serving](https://mlsanity.com/paper/2406.03243/) (2024)
- [Mobile Edge Intelligence for Large Language Models: A Contemporary Survey](https://mlsanity.com/paper/2407.18921/) (2024)
- [Knowledge Fusion of Large Language Models](https://mlsanity.com/paper/2401.10491/) (2024)
- [Modular Pluralism: Pluralistic Alignment via Multi-LLM Collaboration](https://mlsanity.com/paper/2406.15951/) (2024)
- [A Review on Edge Large Language Models: Design, Execution, and Applications](https://mlsanity.com/paper/2410.11845/) (2024)
- [Hymba: A Hybrid-head Architecture for Small Language Models](https://mlsanity.com/paper/2411.13676/) (2024)
- [Harnessing Multiple Large Language Models: A Survey on LLM Ensemble](https://mlsanity.com/paper/2502.18036/) (2025)
- [ReMA: Learning to Meta-think for LLMs with Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2503.09501/) (2025)
- [MixLLM: Dynamic Routing in Mixed Large Language Models](https://mlsanity.com/paper/2502.18482/) (2025)
- [Pack of LLMs: Model Fusion at Test-Time via Perplexity Optimization](https://mlsanity.com/paper/2404.11531/) (2024)
- [Auxiliary task demands mask the capabilities of smaller language models](https://mlsanity.com/paper/2404.02418/) (2024)
- [Personalized Large Language Models](https://mlsanity.com/paper/2402.09269/) (2024)
- [Tiny Machine Learning: Progress and Futures](https://mlsanity.com/paper/2403.19076/) (2024)
- [Long-VITA: Scaling Large Multi-modal Models to 1 Million Tokens with Leading Short-Context Accuracy](https://mlsanity.com/paper/2502.05177/) (2025)
- [Routoo: Learning to Route to Large Language Models Effectively](https://mlsanity.com/paper/2401.13979/) (2024)
- [CoGenesis: A Framework Collaborating Large and Small Language Models for Secure Context-Aware Instruction Following](https://mlsanity.com/paper/2403.03129/) (2024)
- [A Survey of Large Language Model Empowered Agents for Recommendation and Search: Towards Next-Generation Information Retrieval](https://mlsanity.com/paper/2503.05659/) (2025)
- [Smoothie: Label Free Language Model Routing](https://mlsanity.com/paper/2412.04692/) (2024)
- [CITER: Collaborative Inference for Efficient Large Language Model Decoding with Token-Level Routing](https://mlsanity.com/paper/2502.01976/) (2025)
- [MathFusion: Enhancing Mathematical Problem-solving of LLM through Instruction Fusion](https://mlsanity.com/paper/2503.16212/) (2025)
- [Confident or Seek Stronger: Exploring Uncertainty-Based On-device LLM Routing From Benchmarking to Generalization](https://mlsanity.com/paper/2502.04428/) (2025)
- [CE-CoLLM: Efficient and Adaptive Large Language Models Through Cloud-Edge Collaboration](https://mlsanity.com/paper/2411.02829/) (2024)

## Built on by (most cited first, 14 of 14)

- [A Survey of Context Engineering for Large Language Models](https://mlsanity.com/paper/2507.13334/) (2025)
- [GlimpRouter: Efficient Collaborative Inference by Glimpsing One Token of Thoughts](https://mlsanity.com/paper/2601.05110/) (2026)
- [Intelligence per Watt: Measuring Intelligence Efficiency of Local AI](https://mlsanity.com/paper/2511.07885/) (2025)
- [Towards Efficient Multi-LLM Inference: Characterization and Analysis of LLM Routing and Hierarchical Techniques](https://mlsanity.com/paper/2506.06579/) (2025)
- [Small-Large Collaboration: Training-efficient Concept Personalization for Large VLM using a Meta Personalized Small VLM](https://mlsanity.com/paper/2508.07260/) (2025)
- [Triage: Routing Software Engineering Tasks to Cost-Effective LLM Tiers via Code Quality Signals](https://mlsanity.com/paper/2604.07494/) (2026)
- [Uncertainty-Aware Collaborative System of Large and Small Models for Multimodal Sentiment Analysis](https://mlsanity.com/paper/2509.04459/) (2025)
- [RailVQA: A Benchmark and Framework for Efficient Interpretable Visual Cognition in Automatic Train Operation](https://mlsanity.com/paper/2603.27112/) (2026)
- [Benchmarking the Benchmarks: Evaluating Automated Safety Benchmarks for Small Language Models](https://mlsanity.com/paper/2608.17183/) (2026)
- [Adaptive Confidence Gating in Multi-Agent Collaboration for Efficient and Optimized Code Generation](https://mlsanity.com/paper/2601.21469/) (2026)
- [Quantifying Prior Dominance in RAG Systems](https://mlsanity.com/paper/2606.23695/) (2026)
- [MentorPulse: Refreshing Cross-Model Latent Guidance for Long-Form Generation](https://mlsanity.com/paper/2608.20927/) (2026)
- [Learning to Ask: Information Acquisition for SLM-LLM Collaboration, under a budget](https://mlsanity.com/paper/2610.01236/) (2026)
- [Agent in a Bottle: Can LLM Agents Turn Their Capabilities Into Cheap, Scalable Artifacts?](https://mlsanity.com/paper/2610.08775/) (2026)

## Abstract (arXiv)

Large Language Models (LLMs) deliver powerful AI capabilities but face deployment challenges due to high resource costs and latency, whereas Small Language Models (SLMs) offer efficiency and deployability at the cost of reduced performance. Collaboration between LLMs and SLMs emerges as a crucial paradigm to synergistically balance these trade-offs, enabling advanced AI applications, especially on resource-constrained edge devices. This survey provides a comprehensive overview of LLM-SLM collaboration, detailing various interaction mechanisms (pipeline, routing, auxiliary, distillation, fusion), key enabling technologies, and diverse application scenarios driven by on-device needs like low latency, privacy, personalization, and offline operation. While highlighting the significant potential for creating more efficient, adaptable, and accessible AI, we also discuss persistent challenges including system overhead, inter-model consistency, robust task allocation, evaluation complexity, and security/privacy concerns. Future directions point towards more intelligent adaptive frameworks, deeper model fusion, and expansion into multimodal and embodied AI, positioning LLM-SLM collaboration as a key driver for the next generation of practical and ubiquitous artificial intelligence.