mlsanityData to 14 Oct 2026

A Survey on Collaborative Mechanisms Between Large and Small Language Models

arXiv 2505.07460 ↗First posted 12 May 2025cs.AI cs.CLcited by 14

Not on the map. Why?

Abstract (arXiv)

Large Language Models (LLMs) deliver powerful AI capabilities but face deployment challenges due to high resource costs and latency, whereas Small Language Models (SLMs) offer efficiency and deployability at the cost of reduced performance. Collaboration between LLMs and SLMs emerges as a crucial paradigm to synergistically balance these trade-offs, enabling advanced AI applications, especially on resource-constrained edge devices. This survey provides a comprehensive overview of LLM-SLM collaboration, detailing various interaction mechanisms (pipeline, routing, auxiliary, distillation, fusion), key enabling technologies, and diverse application scenarios driven by on-device needs like low latency, privacy, personalization, and offline operation. While highlighting the significant potential for creating more efficient, adaptable, and accessible AI, we also discuss persistent challenges including system overhead, inter-model consistency, robust task allocation, evaluation complexity, and security/privacy concerns. Future directions point towards more intelligent adaptive frameworks, deeper model fusion, and expansion into multimodal and embodied AI, positioning LLM-SLM collaboration as a key driver for the next generation of practical and ubiquitous artificial intelligence.

Key facts. Cited by 14 papers in the map (data through 14 Oct 2026). Builds on 69 of its 93 references that are in the map. First posted on arXiv 12 May 2025.

Builds on

69 of 93 references.

References · 69

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,790
  3. 2023A Comprehensive Overview of Large Language Modelscited by 648
  4. 2024Hybrid LLM: Cost-Efficient and Quality-Aware Query Routingcited by 372
  5. 2024A Survey on Knowledge Distillation of Large Language Modelscited by 284
  6. 2024Model Merging in LLMs, MLLMs, and Beyond: Methods, Theories, Applications and Opportunitiescited by 254
  7. 2024Chain of Agents: Large Language Models Collaborating on Long-Context Taskscited by 206
  8. 2025VITA-1.5: Towards GPT-4o Level Real-Time Vision and Speech Interactioncited by 197
  9. 2024Llumnix: Dynamic Scheduling for Large Language Model Servingcited by 174
  10. 2024Mobile Edge Intelligence for Large Language Models: A Contemporary Surveycited by 128
  11. 2024Knowledge Fusion of Large Language Modelscited by 115
  12. 2024Modular Pluralism: Pluralistic Alignment via Multi-LLM Collaborationcited by 113
38 more
  1. 2024A Review on Edge Large Language Models: Design, Execution, and Applicationscited by 100
  2. 2024Hymba: A Hybrid-head Architecture for Small Language Modelscited by 99
  3. 2025Harnessing Multiple Large Language Models: A Survey on LLM Ensemblecited by 86
  4. 2025ReMA: Learning to Meta-think for LLMs with Multi-Agent Reinforcement Learningcited by 79
  5. 2025MixLLM: Dynamic Routing in Mixed Large Language Modelscited by 59
  6. 2024Pack of LLMs: Model Fusion at Test-Time via Perplexity Optimizationcited by 56
  7. 2024Auxiliary task demands mask the capabilities of smaller language modelscited by 54
  8. 2024Personalized Large Language Modelscited by 44
  9. 2024Tiny Machine Learning: Progress and Futurescited by 40
  10. 2025Long-VITA: Scaling Large Multi-modal Models to 1 Million Tokens with Leading Short-Context Accuracycited by 39
  11. 2024Routoo: Learning to Route to Large Language Models Effectivelycited by 34
  12. 2024CoGenesis: A Framework Collaborating Large and Small Language Models for Secure Context-Aware Instruction Followingcited by 33
  13. 2025A Survey of Large Language Model Empowered Agents for Recommendation and Search: Towards Next-Generation Information Retrievalcited by 33
  14. 2024Smoothie: Label Free Language Model Routingcited by 31
  15. 2025CITER: Collaborative Inference for Efficient Large Language Model Decoding with Token-Level Routingcited by 30
  16. 2025MathFusion: Enhancing Mathematical Problem-solving of LLM through Instruction Fusioncited by 30
  17. 2025Confident or Seek Stronger: Exploring Uncertainty-Based On-device LLM Routing From Benchmarking to Generalizationcited by 29
  18. 2024CE-CoLLM: Efficient and Adaptive Large Language Models Through Cloud-Edge Collaborationcited by 25
  19. 2024A Survey on Large Language Models for Communication, Network, and Service Management: Application Insights, Challenges, and Future Directionscited by 25
  20. 2025Division-of-Thoughts: Harnessing Hybrid Language Model Synergy for Efficient On-Device Agentscited by 25
  21. 2025Multi-Agent Geospatial Copilots for Remote Sensing Workflowscited by 22
  22. 2025Doing More with Less: A Survey on Routing Strategies for Resource Optimisation in Large Language Model-Based Systemscited by 20
  23. 2025Minions: Cost-efficient Collaboration Between On-device and Cloud Language Modelscited by 20
  24. 2025Collab-RAG: Boosting Retrieval-Augmented Generation for Complex Question Answering via White-Box and Black-Box LLM Collaborationcited by 20
  25. 2024SLMRec: Distilling Large Language Models into Small for Sequential Recommendationcited by 18
  26. 2024Bias Amplification in Language Model Evolution: An Iterated Learning Perspectivecited by 17
  27. 2025LLM-Empowered Embodied Agent for Memory-Augmented Task Planning in Household Roboticscited by 17
  28. 2025Collaboration of Large Language Models and Small Recommendation Models for Device-Cloud Recommendationcited by 15
  29. 2025Self-Resource Allocation in Multi-Agent LLM Systemscited by 15
  30. 2023Mutual Enhancement of Large and Small Language Models with Cross-Silo Knowledge Transfercited by 13
  31. 2024Fast and Slow Generating: An Empirical Study on Large and Small Language Models Collaborative Decodingcited by 13
  32. 2025Exploring Embodied Multimodal Large Models: Development, Datasets, and Future Directionscited by 13
  33. 2025Improving Consistency in Large Language Models through Chain of Guidancecited by 13
  34. 2024LLM for Mobile: An Initial Roadmapcited by 11
  35. 2024Enhancing Knowledge Distillation of Large Language Models through Efficient Multi-Modal Distribution Alignmentcited by 11
  36. 2025Life-Cycle Routing Vulnerabilities of LLM Routercited by 11
  37. 2025Hawkeye:Efficient Reasoning with Model Collaborationcited by 11
  38. 2025Dynamic Low-Rank Sparse Adaptation for Large Language Modelscited by 10

50 most cited of 69 All 69 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 5; 2026: 9.

14 papers.

Sort:

By area:

  1. 2025A Survey of Context Engineering for Large Language Modelscited by 126
  2. 2026GlimpRouter: Efficient Collaborative Inference by Glimpsing One Token of Thoughtscited by 17
  3. 2025Intelligence per Watt: Measuring Intelligence Efficiency of Local AIcited by 14
  4. 2025Towards Efficient Multi-LLM Inference: Characterization and Analysis of LLM Routing and Hierarchical Techniquescited by 5
  5. 2025Small-Large Collaboration: Training-efficient Concept Personalization for Large VLM using a Meta Personalized Small VLMcited by 4
  6. 2026Triage: Routing Software Engineering Tasks to Cost-Effective LLM Tiers via Code Quality Signalscited by 3
  7. 2025Uncertainty-Aware Collaborative System of Large and Small Models for Multimodal Sentiment Analysiscited by 1
  8. 2026RailVQA: A Benchmark and Framework for Efficient Interpretable Visual Cognition in Automatic Train Operationcited by 1
  9. 2026Benchmarking the Benchmarks: Evaluating Automated Safety Benchmarks for Small Language Modelscited by 1
  10. 2026Adaptive Confidence Gating in Multi-Agent Collaboration for Efficient and Optimized Code Generationcited by 0
  11. 2026Quantifying Prior Dominance in RAG Systemscited by 0
  12. 2026MentorPulse: Refreshing Cross-Model Latent Guidance for Long-Form Generationcited by 0
2 more
  1. 2026Learning to Ask: Information Acquisition for SLM-LLM Collaboration, under a budgetcited by 0
  2. 2026Agent in a Bottle: Can LLM Agents Turn Their Capabilities Into Cheap, Scalable Artifacts?cited by 0