# Dr.LLM: Dynamic Layer Routing in LLMs

arXiv: https://arxiv.org/abs/2510.12773 · HTML: https://mlsanity.com/paper/2510.12773/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 34 references that are in the map.
- First posted on arXiv 14 Oct 2025.

## Builds on (most cited first, 29 of 29)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [Qwen2.5 Technical Report](https://mlsanity.com/paper/2412.15115/) (2024)
- [Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge](https://mlsanity.com/paper/1803.05457/) (2018)
- [Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer](https://mlsanity.com/paper/1701.06538/) (2017)
- [Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity](https://mlsanity.com/paper/2101.03961/) (2021)
- [TruthfulQA: Measuring How Models Mimic Human Falsehoods](https://mlsanity.com/paper/2109.07958/) (2021)
- [PIQA: Reasoning about Physical Commonsense in Natural Language](https://mlsanity.com/paper/1911.11641/) (2019)
- [GPQA: A Graduate-Level Google-Proof Q&A Benchmark](https://mlsanity.com/paper/2311.12022/) (2023)
- [Know What You Don't Know: Unanswerable Questions for SQuAD](https://mlsanity.com/paper/1806.03822/) (2018)
- [AGIEval: A Human-Centric Benchmark for Evaluating Foundation Models](https://mlsanity.com/paper/2304.06364/) (2023)
- [BranchyNet: Fast Inference via Early Exiting from Deep Neural Networks](https://mlsanity.com/paper/1709.01686/) (2017)
- [Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approach](https://mlsanity.com/paper/2502.05171/) (2025)
- [ShortGPT: Layers in Large Language Models are More Redundant Than You Expect](https://mlsanity.com/paper/2403.03853/) (2024)
- [DeeBERT: Dynamic Early Exiting for Accelerating BERT Inference](https://mlsanity.com/paper/2004.12993/) (2020)
- [LayerSkip: Enabling Early Exit Inference and Self-Speculative Decoding](https://mlsanity.com/paper/2404.16710/) (2024)
- [BERT Loses Patience: Fast and Robust Inference with Early Exit](https://mlsanity.com/paper/2006.04152/) (2020)
- [Mixture-of-Depths: Dynamically allocating compute in transformer-based language models](https://mlsanity.com/paper/2404.02258/) (2024)
- [Looped Transformers as Programmable Computers](https://mlsanity.com/paper/2301.13196/) (2023)
- [Camels in a Changing Climate: Enhancing LM Adaptation with Tulu 2](https://mlsanity.com/paper/2311.10702/) (2023)
- [Looped Transformers are Better at Learning Learning Algorithms](https://mlsanity.com/paper/2311.12424/) (2023)
- [Mixture-of-Recursions: Learning Dynamic Recursive Depths for Adaptive Token-Level Computation](https://mlsanity.com/paper/2507.10524/) (2025)
- [On the Effect of Dropping Layers of Pre-trained Transformer Models](https://mlsanity.com/paper/2004.03844/) (2020)
- [DART-Math: Difficulty-Aware Rejection Tuning for Mathematical Problem-Solving](https://mlsanity.com/paper/2407.13690/) (2024)
- [Calibrating Large Language Models Using Their Generations Only](https://mlsanity.com/paper/2403.05973/) (2024)
- [Adaptive Layer-skipping in Pre-trained LLMs](https://mlsanity.com/paper/2503.23798/) (2025)
- [Skip a Layer or Loop it? Test-Time Depth Adaptation of Pretrained LLMs](https://mlsanity.com/paper/2507.07996/) (2025)
- [Router-Tuning: A Simple and Effective Approach for Enabling Dynamic-Depth in Transformers](https://mlsanity.com/paper/2410.13184/) (2024)

## Built on by (most cited first, 12 of 12)

- [LoopMoE: Unifying Iterative Computation with Mixture-of-Experts for Language Modeling](https://mlsanity.com/paper/2606.04438/) (2026)
- [Skip a Layer or Loop It? Learning Program-of-Layers in LLMs](https://mlsanity.com/paper/2606.06574/) (2026)
- [Reasoning-preserved Efficient Distillation of Large Language Models via Activation-aware Initialization](https://mlsanity.com/paper/2605.29327/) (2026)
- [MACRO: Markov Chain Routing of Transformer Layers](https://mlsanity.com/paper/2608.05872/) (2026)
- [Towards Understanding Layer Contributions in Tabular In-Context Learning Models](https://mlsanity.com/paper/2511.15432/) (2025)
- [Reachability Is Not Realization: Tracing the Sources of LLM Benchmark Gains](https://mlsanity.com/paper/2608.03219/) (2026)
- [Programs-of-Layers in LLMs through the Lens of Cortical Areas](https://mlsanity.com/paper/2609.31360/) (2026)
- [Do Dynamic Routers Need Memory? HeRo: History-Aware Routing for Efficient LLM Inference](https://mlsanity.com/paper/2609.08189/) (2026)
- [When Less Compute Is More: Adaptive Early Exit Improves Pretrained Outlier Detection](https://mlsanity.com/paper/2609.32898/) (2026)
- [Dynamic Routing as a New Dimension for Test-time Versatility of LLMs](https://mlsanity.com/paper/2610.04751/) (2026)
- [Input-Blind Controls Produce Substantial Oracle Headroom for Layer Programs in Multiple-Choice Evaluation](https://mlsanity.com/paper/2610.10368/) (2026)
- [Running Fewer Loops Removes Refusal Behaviour in Recurrent-Depth Language Models](https://mlsanity.com/paper/2610.15218/) (2026)

## Abstract (arXiv)

Large Language Models (LLMs) process every token through all layers of a transformer stack, causing wasted computation on simple queries and insufficient flexibility for harder ones that need deeper reasoning. Adaptive-depth methods can improve efficiency, but prior approaches rely on costly inference-time search, architectural changes, or large-scale retraining, and in practice often degrade accuracy despite efficiency gains. We introduce Dr. LLM, Dynamic routing of Layers for LLMs, a retrofittable framework that equips pretrained models with lightweight per-layer routers deciding to skip, execute, or repeat a block. Routers are trained with explicit supervision: using Monte Carlo Tree Search (MCTS), we derive high-quality layer configurations that preserve or improve accuracy under a compute budget. Our design, windowed pooling for stable routing, focal loss with class balancing, and bottleneck MLP routers, ensures robustness under class imbalance and long sequences. On ARC (logic) and DART (math), Dr. LLM improves accuracy by up to +3.4%p while saving 5 layers per example on average. Routers generalize to out-of-domain tasks (MMLU, GSM8k, AIME, TruthfulQA, SQuADv2, GPQA, PIQA, AGIEval) with only 0.85% accuracy drop while retaining efficiency, and outperform prior routing methods by up to +7.7%p. Overall, Dr. LLM shows that explicitly supervised routers retrofit frozen LLMs for budget-aware, accuracy-driven inference without altering base weights. Code is available at https://github.com/parameterlab/dr-llm.