mlsanityData to 14 Oct 2026

Fusing Models with Complementary Expertise

arXiv 2310.01542 ↗First posted 2 Oct 2023cs.LGcited by 46

Not on the map. Why?

Abstract (arXiv)

Training AI models that generalize across tasks and domains has long been among the open problems driving AI research. The emergence of Foundation Models made it easier to obtain expert models for a given task, but the heterogeneity of data that may be encountered at test time often means that any single expert is insufficient. We consider the Fusion of Experts (FoE) problem of fusing outputs of expert models with complementary knowledge of the data distribution and formulate it as an instance of supervised learning. Our method is applicable to both discriminative and generative tasks and leads to significant performance improvements in image and text classification, text summarization, multiple-choice QA, and automatic evaluation of generated text. We also extend our method to the "frugal" setting where it is desired to reduce the number of expert model evaluations at test time. Our implementation is publicly available at https://github.com/hwang595/FoE-ICLR2024.

Key facts. Cited by 46 papers in the map (data through 14 Oct 2026). Builds on 45 of its 83 references that are in the map. First posted on arXiv 2 Oct 2023.

Builds on

45 of 83 references.

References · 45

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  4. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,669
  5. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,893
  6. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,512
  7. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  8. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,183
  9. 2017On Calibration of Modern Neural Networkscited by 4,960
  10. 2021On the Opportunities and Risks of Foundation Modelscited by 4,033
  11. 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,926
  12. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
33 more
  1. 2019Advances and Open Problems in Federated Learningcited by 3,046
  2. 2020WILDS: A Benchmark of in-the-Wild Distribution Shiftscited by 1,375
  3. 2019PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarizationcited by 1,102
  4. 2020On Faithfulness and Factuality in Abstractive Summarizationcited by 900
  5. 2023FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performancecited by 853
  6. 2017Selective Classification for Deep Neural Networkscited by 852
  7. 2022Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distributioncited by 739
  8. 2023Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizescited by 737
  9. 2021BARTScore: Evaluating Generated Text as Text Generationcited by 722
  10. 2020Ensemble Distillation for Robust Model Fusion in Federated Learningcited by 682
  11. 2020SummEval: Re-evaluating Summarization Evaluationcited by 674
  12. 2023Textbooks Are All You Needcited by 569
  13. 2020Federated Learning with Matched Averagingcited by 564
  14. 2023LLM-Blender: Ensembling Large Language Models with Pairwise Ranking and Generative Fusioncited by 564
  15. 2019FedMD: Heterogenous Federated Learning via Model Distillationcited by 459
  16. 2021Towards Out-Of-Distribution Generalization: A Surveycited by 436
  17. 2020Asking and Answering Questions to Evaluate the Factual Consistency of Summariescited by 378
  18. 2018Detecting and Correcting for Label Shift with Black Box Predictorscited by 366
  19. 2021A Field Guide to Federated Optimizationcited by 315
  20. 2018Newsroom: A Dataset of 1.3 Million Summaries with Diverse Extractive Strategiescited by 294
  21. 2021Ensemble deep learning: A reviewcited by 284
  22. 2022Towards a Unified Multi-Dimensional Evaluator for Text Generationcited by 264
  23. 2023Specializing Smaller Language Models towards Multi-Step Reasoningcited by 251
  24. 2022Out-of-Distribution Detection and Selective Generation for Conditional Language Modelscited by 197
  25. 2020BREEDS: Benchmarks for Subpopulation Shiftcited by 155
  26. 2023Exploring the Benefits of Training Expert Language Models over Instruction Tuningcited by 80
  27. 2021Understanding Softmax Confidence and Uncertaintycited by 63
  28. 2022SummaReranker: A Multi-Task Mixture-of-Experts Re-ranking Framework for Abstractive Summarizationcited by 61
  29. 2020Investigating Societal Biases in a Poetry Composition Systemcited by 48
  30. 2020FrugalML: How to Use ML Prediction APIs More Accurately and Cheaplycited by 33
  31. 2022On the Evaluation Metrics for Paraphrase Generationcited by 32
  32. 2020Generalized Negative Correlation Learning for Deep Ensemblingcited by 17
  33. 2023Confidence-Based Model Selection: When to Take Shortcuts for Subpopulation Shiftscited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 18; 2025: 18; 2026: 10.

46 papers.

Sort:

By area:

  1. 2024Mixture-of-Agents Enhances Large Language Model Capabilitiescited by 438
  2. 2024More Agents Is All You Needcited by 180
  3. 2024Twin-Merging: Dynamic Integration of Modular Expertise in Model Mergingcited by 116
  4. 2024RouterDC: Query-Based Router by Dual Contrastive Learning for Assembling Large Language Modelscited by 88
  5. 2024Ensemble Learning for Heterogeneous Large Language Models with Deep Parallel Collaborationcited by 67
  6. 2024Pack of LLMs: Model Fusion at Test-Time via Perplexity Optimizationcited by 56
  7. 2024MetaLLM: A High-performant and Cost-efficient Dynamic Framework for Wrapping LLMscited by 41
  8. 2025CARROT: A Cost Aware Rate Optimal Routercited by 39
  9. 2025KABB: Knowledge-Aware Bayesian Bandits for Dynamic Expert Coordination in Multi-Agent Systemscited by 37
  10. 2025CITER: Collaborative Inference for Efficient Large Language Model Decoding with Token-Level Routingcited by 30
  11. 2024Determine-Then-Ensemble: Necessity of Top-k Union for Large Language Model Ensemblingcited by 28
  12. 2025Causal LLM Routing: End-to-End Regret Minimization from Observational Datacited by 28
34 more
  1. 2024Model Swarms: Collaborative Search to Adapt LLM Experts via Swarm Intelligencecited by 27
  2. 2024A Survey: Collaborative Hardware and Software Design in the Era of Large Language Modelscited by 23
  3. 2025Doing More with Less: A Survey on Routing Strategies for Resource Optimisation in Large Language Model-Based Systemscited by 20
  4. 2024LLM-based multi-agent poetry generation in non-cooperative environmentscited by 15
  5. 2025Efficient Training-Free Online Routing for High-Volume Multi-LLM Servingcited by 13
  6. 2025Rethinking Predictive Modeling for LLM Routing: When Simple kNN Beats Complex Learned Routerscited by 11
  7. 2025Multi-LLM Collaborative Search for Complex Problem Solvingcited by 10
  8. 2024On Giant's Shoulders: Effortless Weak to Strong by Dynamic Logits Fusioncited by 9
  9. 2025Ensembles of Low-Rank Expert Adapterscited by 9
  10. 2025Ensemble Learning for Large Language Models in Text and Code Generation: A Surveycited by 9
  11. 2025Speculate, then Collaborate: Fusing Knowledge of Language Models during Decodingcited by 8
  12. 2024Multilingual Arbitrage: Optimizing Data Pools to Accelerate Multilingual Progresscited by 7
  13. 2024Copyright-Protected Language Generation via Adaptive Model Fusioncited by 7
  14. 2024Efficiently Deploying LLMs with Controlled Riskcited by 5
  15. 2024Block Selective Reprogramming for On-device Training of Vision Transformerscited by 4
  16. 2024Strong Copyright Protection for Language Models via Adaptive Model Fusioncited by 4
  17. 2026AdaFuse: Adaptive Ensemble Decoding with Test-Time Scaling for LLMscited by 4
  18. 2025Near-Optimal Online Deployment and Routing for Streaming LLMscited by 3
  19. 2026PPAI: Enabling Personalized LLM Agent Interoperability for Collaborative Edge Intelligencecited by 3
  20. 2026Mixture of Complementary Agents for Robust LLM Ensemblecited by 3
  21. 2025Reasoning-Intensive Regressioncited by 2
  22. 2025Robust Federated Inferencecited by 2
  23. 2024Brainstorming Brings Power to Large Language Models of Knowledge Reasoningcited by 1
  24. 2025LightRouter: Towards Efficient LLM Collaboration with Minimal Overheadcited by 1
  25. 2025Enabling Flexible Multi-LLM Integration for Scalable Knowledge Aggregationcited by 1
  26. 2025MCCE: A Framework for Multi-LLM Collaborative Search in Discrete Spaces with Similarity-Filtered Preference Learningcited by 1
  27. 2026RankGuide: Tensor-Rank-Guided Routing and Steering for Efficient Reasoningcited by 1
  28. 2026A Red Teaming Framework for Large Language Models: A Case Study on Faithfulness Evaluationcited by 1
  29. 2026Training-Free versus Training-Based Intent Classification in LLMs: Accuracy, Robustness, and Failure Modescited by 1
  30. 2025LeForecast: Enterprise Hybrid Forecast by Time Series Intelligencecited by 0
  31. 2026n-Musketeers: Reinforcement Learning Shapes Collaboration Among Language Modelscited by 0
  32. 2026Trade-offs in Ensembling, Merging and Routing Among Parameter-Efficient Expertscited by 0
  33. 2026Capturing LLM Capabilities via Evidence-Calibrated Query Clusteringcited by 0
  34. 2026Token-Operations-Oriented Inference Optimization Techniques for Large Modelscited by 0