# ICL-Router: In-Context Learned Model Representations for LLM Routing

arXiv: https://arxiv.org/abs/2510.09719 · HTML: https://mlsanity.com/paper/2510.09719/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 36 references that are in the map.
- First posted on arXiv 10 Oct 2025.

## Builds on (most cited first, 26 of 26)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Program Synthesis with Large Language Models](https://mlsanity.com/paper/2108.07732/) (2021)
- [MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark](https://mlsanity.com/paper/2406.01574/) (2024)
- [Gemma 3 Technical Report](https://mlsanity.com/paper/2503.19786/) (2025)
- [Qwen3 Embedding: Advancing Text Embedding and Reranking Through Foundation Models](https://mlsanity.com/paper/2506.05176/) (2025)
- [M3-Embedding: Multi-Linguality, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillation](https://mlsanity.com/paper/2402.03216/) (2024)
- [Towards General Text Embeddings with Multi-stage Contrastive Learning](https://mlsanity.com/paper/2308.03281/) (2023)
- [Long-context LLMs Struggle with Long In-context Learning](https://mlsanity.com/paper/2404.02060/) (2024)
- [In-context Vectors: Making In Context Learning More Effective and Controllable Through Latent Space Steering](https://mlsanity.com/paper/2311.06668/) (2023)
- [General-Reasoner: Advancing LLM Reasoning Across All Domains](https://mlsanity.com/paper/2505.14652/) (2025)
- [In-Context Learning with Long-Context Models: An In-Depth Exploration](https://mlsanity.com/paper/2405.00200/) (2024)
- [Jasper and Stella: distillation of SOTA embedding models](https://mlsanity.com/paper/2412.19048/) (2024)
- [Gold-medalist Performance in Solving Olympiad Geometry with AlphaGeometry2](https://mlsanity.com/paper/2502.03544/) (2025)
- [Universal Model Routing for Efficient LLM Inference](https://mlsanity.com/paper/2502.08773/) (2025)
- [AceMath: Advancing Frontier Math Reasoning with Post-Training and Reward Modeling](https://mlsanity.com/paper/2412.15084/) (2024)
- [AceReason-Nemotron 1.1: Advancing Math and Code Reasoning through SFT and RL Synergy](https://mlsanity.com/paper/2506.13284/) (2025)
- [KOR-Bench: Benchmarking Language Models on Knowledge-Orthogonal Reasoning Tasks](https://mlsanity.com/paper/2410.06526/) (2024)
- [How Many Demonstrations Do You Need for In-context Learning?](https://mlsanity.com/paper/2303.08119/) (2023)
- [MMLU-CF: A Contamination-free Multi-task Language Understanding Benchmark](https://mlsanity.com/paper/2412.15194/) (2024)
- [In-Context Learning with Many Demonstration Examples](https://mlsanity.com/paper/2302.04931/) (2023)
- [The Avengers: A Simple Recipe for Uniting Smaller Language Models to Challenge Proprietary Giants](https://mlsanity.com/paper/2505.19797/) (2025)
- [Capability Instruction Tuning: A New Paradigm for Dynamic LLM Routing](https://mlsanity.com/paper/2502.17282/) (2025)
- [Vector-ICL: In-context Learning with Continuous Vector Representations](https://mlsanity.com/paper/2410.05629/) (2024)
- [More is not always better? Enhancing Many-Shot In-Context Learning with Differentiated and Reweighting Objectives](https://mlsanity.com/paper/2501.04070/) (2025)
- [SHARP: Synthesizing High-quality Aligned Reasoning Problems for Large Reasoning Models Reinforcement Learning](https://mlsanity.com/paper/2505.14147/) (2025)
- [Thought-Path Contrastive Learning via Premise-Oriented Data Augmentation for Logical Reading Comprehension](https://mlsanity.com/paper/2409.14495/) (2024)

## Built on by (most cited first, 28 of 28)

- [LLMRouterBench: A Massive Benchmark and Unified Framework for LLM Routing](https://mlsanity.com/paper/2601.07206/) (2026)
- [Dynamic Model Routing and Cascading for Efficient LLM Inference: A Survey](https://mlsanity.com/paper/2603.04445/) (2026)
- [Atlas: Orchestrating Heterogeneous Models and Tools for Multi-Domain Complex Reasoning](https://mlsanity.com/paper/2601.03872/) (2026)
- [When Routing Collapses: On the Degenerate Convergence of LLM Routers](https://mlsanity.com/paper/2602.03478/) (2026)
- [Effective LoRA Adapter Routing using Task Representations](https://mlsanity.com/paper/2601.21795/) (2026)
- [RoboRouter: Training-Free Policy Routing for Robotic Manipulation](https://mlsanity.com/paper/2603.07892/) (2026)
- [Uno-Orchestra: Parsimonious Agent Routing via Selective Delegation](https://mlsanity.com/paper/2605.05007/) (2026)
- [MonoScale: Scaling Multi-Agent System with Monotonic Improvement](https://mlsanity.com/paper/2601.23219/) (2026)
- [Route to Rome Attack: Directing LLM Routers to Expensive Models via Adversarial Suffix Optimization](https://mlsanity.com/paper/2604.15022/) (2026)
- [Holos: A Web-Scale LLM-Based Multi-Agent System for the Agentic Web](https://mlsanity.com/paper/2604.02334/) (2026)
- [Switchcraft: AI Model Router for Agentic Tool Calling](https://mlsanity.com/paper/2605.07112/) (2026)
- [SeqRoute: Global Budget-Aware Sequential LLM Routing via Offline Reinforcement Learning](https://mlsanity.com/paper/2605.25424/) (2026)
- [FlyRoute: Self-Evolving Agent Profiling via Data Flywheel for Adaptive Task Routing](https://mlsanity.com/paper/2605.22057/) (2026)
- [RouteJudge: An Open Platform for Reproducible and Preference-Aware LLM Routing](https://mlsanity.com/paper/2606.18774/) (2026)
- [VDAR-Router: Adaptive LLMs Routing via Verbalized Query Difficulty Analysis Retrieval](https://mlsanity.com/paper/2607.18098/) (2026)
- [Error-Aware Reverse Auction Mechanism for Large Language Model Routing](https://mlsanity.com/paper/2608.12719/) (2026)
- [RSI-Router: Evolving Subtask-Level LLM Routing and Skills for Cost-Efficient Agents](https://mlsanity.com/paper/2609.34712/) (2026)
- [From Topology to Trajectory: LLM-Driven World Models For Supply Chain Resilience](https://mlsanity.com/paper/2604.11041/) (2026)
- [Beyond Query Memorization: Large Language Model Routing with Query Decomposition and Historical Matching](https://mlsanity.com/paper/2605.25558/) (2026)
- [IR3DE: A Linear Router for Large Language Models](https://mlsanity.com/paper/2606.06098/) (2026)
- [Learning Compositional Meta-Routing for Agentic Workflows: An Executable Benchmark](https://mlsanity.com/paper/2608.00106/) (2026)
- [SCOPE-Router: Cost-Aware Open-Set VLM Routing for Execution-Oriented Tasks](https://mlsanity.com/paper/2608.12127/) (2026)
- [CERA-MoA: Co-Evolving Routing Mechanisms with Continually Learning LLM Agents](https://mlsanity.com/paper/2609.18779/) (2026)
- [Routing Without Embeddings: Fast And Interpretable Routing With Regular Expressions](https://mlsanity.com/paper/2609.34326/) (2026)
- [SeLMRoute: Probabilistic Semantic Evidence for Large Language Model Routing](https://mlsanity.com/paper/2609.34736/) (2026)
- [Pretrain Once, Route Anywhere: Towards a Foundation Model for LLM Routing](https://mlsanity.com/paper/2609.37362/) (2026)
- [MHR-Router: Multimodal Heterogeneous Retrieval-Enhanced Adaptive Routing for VLM Inference](https://mlsanity.com/paper/2610.13278/) (2026)
- [RouteWeaver: Weaving Mode Selection and Execution into Unified LLM Routing](https://mlsanity.com/paper/2610.19705/) (2026)

## Abstract (arXiv)

Large language models (LLMs) often exhibit complementary strengths. Model routing harnesses these strengths by dynamically directing each query to the most suitable model, given a candidate model pool. However, routing performance relies on accurate model representations, and adding new models typically requires retraining, limiting scalability. To address these challenges, we propose a novel routing method using in-context vectors to represent model capabilities. The method proceeds in two stages. First, queries are embedded and projected into vectors, with a projector and LLM-based router trained to reconstruct the original queries, aligning vector representations with the router's semantic space. Second, each candidate model is profiled on a query set, and the router learns -- based on in-context vectors of query and model performance -- to predict whether each model can correctly answer new queries. Extensive experiments demonstrate that our method achieves state-of-the-art routing performance in both in-distribution and out-of-distribution tasks. Moreover, our method allows for seamless integration of new models without retraining the router. The code is available at https://github.com/lalalamdbf/ICL-Router.