mlsanityData to 14 Oct 2026

RouterDC: Query-Based Router by Dual Contrastive Learning for Assembling Large Language Models

arXiv 2409.19886 ↗First posted 30 Sep 2024cs.LG cs.AI cs.CLcited by 88

Not on the map. Why?

Abstract (arXiv)

Recent works show that assembling multiple off-the-shelf large language models (LLMs) can harness their complementary abilities. To achieve this, routing is a promising method, which learns a router to select the most suitable LLM for each query. However, existing routing models are ineffective when multiple LLMs perform well for a query. To address this problem, in this paper, we propose a method called query-based Router by Dual Contrastive learning (RouterDC). The RouterDC model consists of an encoder and LLM embeddings, and we propose two contrastive learning losses to train the RouterDC model. Experimental results show that RouterDC is effective in assembling LLMs and largely outperforms individual top-performing LLMs as well as existing routing methods on both in-distribution (+2.76%) and out-of-distribution (+1.90%) tasks. Source code is available at https://github.com/shuhao02/RouterDC.

Key facts. Cited by 88 papers in the map (data through 14 Oct 2026). Builds on 49 of its 59 references that are in the map. First posted on arXiv 30 Sep 2024.

Builds on

49 of 59 references.

References · 49

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2024The Llama 3 Herd of Modelscited by 14,715
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  4. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  5. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  6. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  7. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  8. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  9. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,513
  10. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
  11. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  12. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
37 more
  1. 2021Program Synthesis with Large Language Modelscited by 3,637
  2. 2023Mistral 7Bcited by 3,164
  3. 2021Learning to Prompt for Vision-Language Modelscited by 2,281
  4. 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,181
  5. 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,173
  6. 2023Code Llama: Open Foundation Models for Codecited by 2,139
  7. 2021DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharingcited by 1,254
  8. 2021Unsupervised Dense Information Retrieval with Contrastive Learningcited by 1,216
  9. 2022Editing Models with Task Arithmeticcited by 1,120
  10. 2023FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performancecited by 853
  11. 2023MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Modelscited by 735
  12. 2023C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Modelscited by 708
  13. 2023WizardMath: Empowering Mathematical Reasoning for Large Language Models via Reinforced Evol-Instructcited by 625
  14. 2021Merging Models with Fisher-Weighted Averagingcited by 621
  15. 2023LLM-Blender: Ensembling Large Language Models with Pairwise Ranking and Generative Fusioncited by 564
  16. 2024Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Modelscited by 557
  17. 2023Zephyr: Direct Distillation of LM Alignmentcited by 506
  18. 2021Sentence-T5: Scalable Sentence Encoders from Pre-trained Text-to-Text Modelscited by 463
  19. 2023CMMLU: Measuring massive multitask language understanding in Chinesecited by 449
  20. 2022Dataless Knowledge Fusion by Merging Weights of Language Modelscited by 387
  21. 2024Hybrid LLM: Cost-Efficient and Quality-Aware Query Routingcited by 372
  22. 2023CodeGeeX: A Pre-Trained Model for Code Generation with Multilingual Benchmarking on HumanEval-Xcited by 300
  23. 2024RouterBench: A Benchmark for Multi-LLM Routing Systemcited by 238
  24. 2020CLEAR: Contrastive Learning for Sentence Representationcited by 186
  25. 2024More Agents Is All You Needcited by 180
  26. 2023Large Language Model Routing with Benchmark Datasetscited by 172
  27. 2023Routing to the Expert: Efficient Reward-guided Ensemble of Large Language Modelscited by 145
  28. 2023Large Language Model Cascades with Mixture of Thoughts Representations for Cost-efficient Reasoningcited by 143
  29. 2024Language Model Cascades: Token-level uncertainty and beyondcited by 121
  30. 2024LoraRetriever: Input-Aware LoRA Retrieval and Composition for Mixed Tasks in the Wildcited by 68
  31. 2023Fusing Models with Complementary Expertisecited by 46
  32. 2024Harnessing the Power of Multiple Minds: Lessons Learned from LLM Routingcited by 46
  33. 2023Forward-Backward Reasoning in Large Language Models for Mathematical Verificationcited by 40
  34. 2024GITA: Graph to Visual and Textual Integration for Vision-Language Graph Reasoningcited by 33
  35. 2024Online Cascade Learning for Efficient Inference over Streamscited by 20
  36. 2022Ranking-Enhanced Unsupervised Sentence Representation Learningcited by 15
  37. 2023Effective Structured Prompting by Meta-Learning and Representative Verbalizercited by 14

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 39; 2026: 48.

88 papers.

Sort:

By area:

  1. 2025Universal Model Routing for Efficient LLM Inferencecited by 86
  2. 2025ReMA: Learning to Meta-think for LLMs with Multi-Agent Reinforcement Learningcited by 79
  3. 2025Router-R1: Teaching LLMs Multi-Round Routing and Aggregation via Reinforcement Learningcited by 72
  4. 2025SciReplicate-Bench: Benchmarking LLMs in Agent-driven Algorithmic Reproduction from Research Paperscited by 43
  5. 2025The Avengers: A Simple Recipe for Uniting Smaller Language Models to Challenge Proprietary Giantscited by 38
  6. 2025OmniRouter: Budget and Performance Controllable Multi-LLM Routingcited by 35
  7. 2025Beyond GPT-5: Making LLMs Cheaper and Better via Performance-Efficiency Optimized Routingcited by 34
  8. 2025Prompt-to-Leaderboardcited by 30
  9. 2025Route to Reason: Adaptive Routing for LLM and Reasoning Strategy Selectioncited by 23
  10. 2025Learning to Orchestrate Agents in Natural Language with the Conductorcited by 23
  11. 2026SkillOrchestra: Learning to Route Agents via Skill Transfercited by 23
  12. 2025RouterArena: An Open Platform for Comprehensive Comparison of LLM Routerscited by 22
38 more
  1. 2025Doing More with Less: A Survey on Routing Strategies for Resource Optimisation in Large Language Model-Based Systemscited by 20
  2. 2026Dynamic Model Routing and Cascading for Efficient LLM Inference: A Surveycited by 19
  3. 2025Cost-Aware Contrastive Routing for LLMscited by 16
  4. 2024Optimal Query Allocation in Extractive QA with LLMs: A Learning-to-Defer Framework with Theoretical Guaranteescited by 15
  5. 2025Nature-Inspired Population-Based Evolution of Large Language Modelscited by 15
  6. 2025Learning to Route LLMs from Bandit Feedback: One Policy, Many Trade-offscited by 15
  7. 2026EvolveRouter: Co-Evolving Routing and Prompt for Multi-Agent Question Answeringcited by 15
  8. 2025Intelligence per Watt: Measuring Intelligence Efficiency of Local AIcited by 14
  9. 2025EllieSQL: Cost-Efficient Text-to-SQL with Complexity-Aware Routingcited by 13
  10. 2025Route-and-Reason: Scaling Large Language Model Reasoning with Reinforced Model Routercited by 13
  11. 2026MMR-Bench: A Comprehensive Benchmark for Multimodal LLM Routingcited by 13
  12. 2025Life-Cycle Routing Vulnerabilities of LLM Routercited by 11
  13. 2025Rethinking Predictive Modeling for LLM Routing: When Simple kNN Beats Complex Learned Routerscited by 11
  14. 2025Learning Compact Representations of LLM Abilities via Item Response Theorycited by 10
  15. 2025MESS+: Dynamically Learned Inference-Time LLM Routing in Model Zoos with Service Level Guaranteescited by 9
  16. 2025Arch-Router: Aligning LLM Routing with Human Preferencescited by 9
  17. 2026Exploring Knowledge Purification in Multi-Teacher Knowledge Distillation for LLMscited by 9
  18. 2026Adaptive Collaboration with Humans: Metacognitive Policy Optimization for Multi-Agent LLMs with Continual Learningcited by 9
  19. 2025Challenges in Testing Large Language Model Based Software: A Faceted Taxonomycited by 8
  20. 2025RAGRouter: Learning to Route Queries to Multiple Retrieval-Augmented Language Modelscited by 8
  21. 2025MetaDefense: Defending Finetuning-based Jailbreak Attack Before and During Generationcited by 8
  22. 2025VL-RouterBench: A Benchmark for Vision-Language Model Routingcited by 8
  23. 2026V0: A Generalist Value Model for Any Policy at State Zerocited by 7
  24. 2025Talk to Right Specialists: Iterative Routing in Multi-agent Systems for Question Answeringcited by 6
  25. 2025Slm-mux: Orchestrating small language models for reasoningcited by 6
  26. 2025ProxRouter: Proximity-Weighted LLM Query Routing for Improved Robustness to Outlierscited by 6
  27. 2026RAGRouter-Bench: A Dataset and Benchmark for Adaptive RAG Routingcited by 6
  28. 2026vLLM Semantic Router: Signal Driven Decision Routing for Mixture-of-Modality Modelscited by 6
  29. 2026Uno-Orchestra: Parsimonious Agent Routing via Selective Delegationcited by 6
  30. 2025TRINITY: An Evolved LLM Coordinatorcited by 5
  31. 2025PersonalizedRouter: Personalized LLM Routing via Graph-based User Preference Modelingcited by 4
  32. 2026Prompt-Dependent Ranking of Large Language Models with Uncertainty Quantificationcited by 4
  33. 2026Sakana Fugu Technical Reportcited by 4
  34. 2026LLMRouter: Unified Infrastructure for Developing, Evaluating, and Deploying LLM Routerscited by 4
  35. 2025Near-Optimal Online Deployment and Routing for Streaming LLMscited by 3
  36. 2025Towards Generalized Routing: Model and Agent Orchestration for Adaptive and Efficient Inferencecited by 3
  37. 2025LLMLogAnalyzer: A Clustering-Based Log Analysis Chatbot using Large Language Modelscited by 3
  38. 2026Efficient and Interpretable Multi-Agent LLM Routing via Ant Colony Optimizationcited by 3

50 most cited of 88 All 88 (JSON)