mlsanityData to 14 Oct 2026

Learning to Decode Collaboratively with Multiple Language Models

arXiv 2403.03870 ↗First posted 6 Mar 2024cs.CL cs.LGcited by 62

Not on the map. Why?

Abstract (arXiv)

We propose a method to teach multiple large language models (LLM) to collaborate by interleaving their generations at the token level. We model the decision of which LLM generates the next token as a latent variable. By optimizing the marginal likelihood of a training set under our latent variable model, the base LLM automatically learns when to generate itself and when to call on one of the ``assistant'' language models to generate, all without direct supervision. Token-level collaboration during decoding allows for a fusion of each model's expertise in a manner tailored to the specific task at hand. Our collaborative decoding is especially useful in cross-domain settings where a generalist base LLM learns to invoke domain expert models. On instruction-following, domain-specific QA, and reasoning tasks, we show that the performance of the joint system exceeds that of the individual models. Through qualitative analysis of the learned latent decisions, we show models trained with our method exhibit several interesting collaboration patterns, e.g., template-filling. Our code is available at https://github.com/clinicalml/co-llm.

Key facts. Cited by 62 papers in the map (data through 14 Oct 2026). Builds on 29 of its 45 references that are in the map. First posted on arXiv 6 Mar 2024.

Builds on

29 of 45 references.

References · 29

  1. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  2. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  3. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  4. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  5. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,182
  6. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,509
  7. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,418
  8. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,598
  9. 2023ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIscited by 1,947
  10. 2023Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflectioncited by 1,839
  11. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,745
  12. 2022Fast Inference from Transformers via Speculative Decodingcited by 1,653
17 more
  1. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,392
  2. 2022Atlas: Few-shot Learning with Retrieval Augmented Language Modelscited by 952
  3. 2024DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Modelscited by 867
  4. 2022Mixture-of-Experts with Expert Choice Routingcited by 623
  5. 2022Contrastive Decoding: Open-ended Text Generation as Optimizationcited by 609
  6. 2021DExperts: Decoding-Time Controlled Text Generation with Experts and Anti-Expertscited by 441
  7. 2023How Far Can Camels Go? Exploring the State of Instruction Tuning on Open Resourcescited by 408
  8. 2021FUDGE: Controlled Text Generation With Future Discriminatorscited by 376
  9. 2020Consistent Estimators for Learning to Defer to an Expertcited by 271
  10. 2023Camels in a Changing Climate: Enhancing LM Adaptation with Tulu 2cited by 229
  11. 2022Branch-Train-Merge: Embarrassingly Parallel Training of Expert Language Modelscited by 191
  12. 2024OpenMoE: An Early Effort on Open Mixture-of-Experts Language Modelscited by 187
  13. 2023Who Should Predict? Exact Algorithms For Learning to Defer to Humanscited by 70
  14. 2024Learning to Route Among Specialized Experts for Zero-Shot Generalizationcited by 65
  15. 2023Learning to Reject with a Fixed Predictor: Application to Decontextualizationcited by 32
  16. 2023CombLM: Adapting Black-Box Language Models through Small Fine-Tuned Modelscited by 26
  17. 2019Domain Adaptive Inference for Neural Machine Translationcited by 10

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 17; 2025: 26; 2026: 19.

62 papers.

Sort:

By area:

  1. 2025Harnessing Multiple Large Language Models: A Survey on LLM Ensemblecited by 86
  2. 2024Towards Modular LLMs by Building and Reusing a Library of LoRAscited by 72
  3. 2025Cache-to-Cache: Direct Semantic Communication Between Large Language Modelscited by 61
  4. 2024A Survey on Model MoErging: Recycling and Routing Among Specialized Experts for Collaborative Learningcited by 52
  5. 2025Universal Cross-Tokenizer Distillation via Approximate Likelihood Matchingcited by 41
  6. 2025The Avengers: A Simple Recipe for Uniting Smaller Language Models to Challenge Proprietary Giantscited by 38
  7. 2025Beyond GPT-5: Making LLMs Cheaper and Better via Performance-Efficiency Optimized Routingcited by 34
  8. 2024Smoothie: Label Free Language Model Routingcited by 31
  9. 2025CITER: Collaborative Inference for Efficient Large Language Model Decoding with Token-Level Routingcited by 30
  10. 2024Model Swarms: Collaborative Search to Adapt LLM Experts via Swarm Intelligencecited by 27
  11. 2024Nudging: Inference-time Alignment of LLMs via Guided Decodingcited by 23
  12. 2025Heterogeneous Swarms: Jointly Optimizing Model Roles and Weights for Multi-LLM Systemscited by 23
38 more
  1. 2024DAM: Dynamic Adapter Merging for Continual Video QA Learningcited by 19
  2. 2024A Systematic Assessment of OpenAI o1-Preview for Higher Order Thinking in Educationcited by 16
  3. 2025Communicating Activations Between Language Model Agentscited by 16
  4. 2025Collab: Controlled Decoding using Mixture of Agents for LLM Alignmentcited by 14
  5. 2025Guiding Reasoning in Small Language Models with LLM Assistancecited by 14
  6. 2024Fast and Slow Generating: An Empirical Study on Large and Small Language Models Collaborative Decodingcited by 13
  7. 2024RouterRetriever: Routing over a Mixture of Expert Embedding Modelscited by 12
  8. 2025Speeding up Speculative Decoding via Sequential Approximate Verificationcited by 11
  9. 2025Don't Throw Away Your Pretrained Modelcited by 10
  10. 2024On Giant's Shoulders: Effortless Weak to Strong by Dynamic Logits Fusioncited by 9
  11. 2026Token-Level LLM Collaboration via FusionRoutecited by 9
  12. 2025Speculate, then Collaborate: Fusing Knowledge of Language Models during Decodingcited by 8
  13. 2025What makes Reasoning Models Different? Follow the Reasoning Leader for Efficient Decodingcited by 8
  14. 2026The Consensus Trap: Rescuing Multi-Agent LLMs from Adversarial Majorities via Token-Level Collaborationcited by 8
  15. 2024Temperature-Centric Investigation of Speculative Decoding with Knowledge Distillationcited by 7
  16. 2026MoCo: A One-Stop Shop for Model Collaboration Researchcited by 6
  17. 2024Multi-trait User Simulation with Adaptive Decoding for Conversational Task Assistantscited by 5
  18. 2025CORE: Measuring Multi-Agent LLM Interaction Quality under Game-Theoretic Pressurescited by 5
  19. 2024e-Health CSIRO at "Discharge Me!" 2024: Generating Discharge Summary Sections with Fine-tuned Language Modelscited by 4
  20. 2024Mind Scramble: Unveiling Large Language Model Psychology Via Typoglycemiacited by 4
  21. 2024Glider: Global and Local Instruction-Driven Expert Routercited by 4
  22. 2025Token Level Routing Inference System for Edge Devicescited by 4
  23. 2025EnsemW2S: Enhancing Weak-to-Strong Generalization with Large Language Model Ensemblescited by 4
  24. 2026KV Cache Translation across Heterogeneous Large Language Modelscited by 4
  25. 2025COSMosFL: Ensemble of Small Language Models for Fault Localisationcited by 3
  26. 2025Token-level Ensembling of Models with Different Vocabulariescited by 3
  27. 2025When to Ensemble: Identifying Token-Level Points for Stable and Fast LLM Ensemblingcited by 3
  28. 2025Entropy-Aware Token Rejection for Improving Speculative Decodingcited by 3
  29. 2025Synergistic Weak-Strong Collaboration by Aligning Preferencescited by 2
  30. 2025Dynamic Collaboration of Multi-Language Models based on Minimal Complete Semantic Unitscited by 2
  31. 2026The Single-Multi Evolution Loop for Self-Improving Model Collaboration Systemscited by 2
  32. 2026MentorCollab: Large-to-Small Inference-Time Mentorship for Concise Reasoning in Language Modelscited by 2
  33. 2026The Bicameral Model: Bidirectional Hidden-State Coupling Between Parallel Language Modelscited by 2
  34. 2026Scaling Participation in Modular AI Systemscited by 2
  35. 2026Multi-Head Latent Control: A Unified Interface for LLM Agent Decision Makingcited by 2
  36. 2026Don't Solve, Just Compare: Tiny Advisors for Runtime Intervention in LLM Agentscited by 2
  37. 2024EnsemW2S: Enhancing Weak-to-Strong Generalization with Large Language Model Ensemblescited by 1
  38. 2025Scalable, Symbiotic, AI and Non-AI Agent Based Parallel Discrete Event Simulationscited by 1

50 most cited of 62 All 62 (JSON)