mlsanityData to 7 Oct 2026

Bench-CoE: a Framework for Collaboration of Experts from Benchmark

arXiv 2412.04167 ↗First posted 5 Dec 2024cs.AIcited by 7

Not on the map. Why?

Abstract (arXiv)

Large Language Models (LLMs) are key technologies driving intelligent systems to handle multiple tasks. To meet the demands of various tasks, an increasing number of LLMs-driven experts with diverse capabilities have been developed, accompanied by corresponding benchmarks to evaluate their performance. This paper proposes the Bench-CoE framework, which enables Collaboration of Experts (CoE) by effectively leveraging benchmark evaluations to achieve optimal performance across various tasks. Bench-CoE includes a set of expert models, a router for assigning tasks to corresponding experts, and a benchmark dataset for training the router. Moreover, we formulate Query-Level and Subject-Level approaches based on our framework, and analyze the merits and drawbacks of these two approaches. Finally, we conduct a series of experiments with vary data distributions on both language and multimodal tasks to validate that our proposed Bench-CoE outperforms any single model in terms of overall performance. We hope this method serves as a baseline for further research in this area. The code is available at urlhttps://github.com/ZhangXJ199/Bench-CoE.

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Builds on 27 of its 37 references that are in the map. First posted on arXiv 5 Dec 2024.

Builds on

27 of 37 references.

References · 27

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  4. 2023Visual Instruction Tuningcited by 7,792
  5. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  6. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  7. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  8. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,669
  9. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,534
  10. 2024Qwen2 Technical Reportcited by 2,076
  11. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,044
  12. 2015A large annotated corpus for learning natural language inferencecited by 1,837
15 more
  1. 2024Mixtral of Expertscited by 1,785
  2. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,767
  3. 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,758
  4. 2024MiniCPM-V: A GPT-4V Level MLLM on Your Phonecited by 1,061
  5. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suitescited by 1,033
  6. 2024Are We on the Right Way for Evaluating Large Vision-Language Models?cited by 893
  7. 2023FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performancecited by 791
  8. 2024RouteLLM: Learning to Route LLMs with Preference Datacited by 633
  9. 2023LLM-Blender: Ensembling Large Language Models with Pairwise Ranking and Generative Fusioncited by 550
  10. 2024Hybrid LLM: Cost-Efficient and Quality-Aware Query Routingcited by 354
  11. 2024Smaug: Fixing Failure Modes of Preference Optimisation with DPO-Positivecited by 239
  12. 2024GraphRouter: A Graph-based Router for LLM Selectionscited by 154
  13. 2024Eagle: Efficient Training-Free Router for Multi-LLM Inferencecited by 34
  14. 2024VCR: A Task for Pixel-Level Complex Reasoning in Vision Language Models via Restoring Occluded Textcited by 15
  15. 2024HDMoLE: Mixture of LoRA Experts with Hierarchical Routing and Dynamic Thresholds for Fine-Tuning LLM-based ASR Modelscited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 5; 2026: 2.

7 papers.

  1. 2025Harnessing Multiple Large Language Models: A Survey on LLM Ensemblecited by 86
  2. 2025Doing More with Less: A Survey on Routing Strategies for Resource Optimisation in Large Language Model-Based Systemscited by 18
  3. 2025CoServe: Efficient Collaboration-of-Experts (CoE) Model Inference with Limited Memorycited by 5
  4. 2025CARGO: A Framework for Confidence-Aware Routing of Large Language Modelscited by 3
  5. 2025CoE-Ops: Collaboration of LLM-based Experts for AIOps Question-Answeringcited by 0
  6. 2026DLLG: Dynamic Logit-Level Gating of LLM Expertscited by 0
  7. 2026Mo' Models, Mo' Problems: How to best select model pools when designing Multi-Agent Systemscited by 0