# Entropy Adaptive Decoding: Dynamic Model Switching for Efficient Inference

arXiv: https://arxiv.org/abs/2502.06833 · HTML: https://mlsanity.com/paper/2502.06833/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 6 of its 9 references that are in the map.
- First posted on arXiv 5 Feb 2025.

## Builds on (most cited first, 6 of 6)

- [Measuring Mathematical Problem Solving With the MATH Dataset](https://mlsanity.com/paper/2103.03874/) (2021)
- [The Curious Case of Neural Text Degeneration](https://mlsanity.com/paper/1904.09751/) (2019)
- [Fast Inference from Transformers via Speculative Decoding](https://mlsanity.com/paper/2211.17192/) (2022)
- [Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads](https://mlsanity.com/paper/2401.10774/) (2024)
- [EE-LLM: Large-Scale Training and Inference of Early-Exit Large Language Models with 3D Parallelism](https://mlsanity.com/paper/2312.04916/) (2023)
- [MoDEM: Mixture of Domain Expert Models](https://mlsanity.com/paper/2410.07490/) (2024)

## Built on by (most cited first, 5 of 5)

- [Scaling Test-Time Inference with Policy-Optimized, Dynamic Retrieval-Augmented Generation via KV Caching and Decoding](https://mlsanity.com/paper/2504.01281/) (2025)
- [Entropy-Aware Token Rejection for Improving Speculative Decoding](https://mlsanity.com/paper/2512.23765/) (2025)
- [Spectral Logit Sculpting: Adaptive Low-Rank Logit Transformation for Controlled Text Generation](https://mlsanity.com/paper/2509.25204/) (2025)
- [Entropy-Guided k-Guard Sampling for Long-Horizon Autoregressive Video Generation](https://mlsanity.com/paper/2601.19488/) (2026)
- [Entropy-informed Decoding: Adaptive Information-Driven Branching](https://mlsanity.com/paper/2605.09745/) (2026)

## Abstract (arXiv)

We present Entropy Adaptive Decoding (EAD), a novel approach for efficient language model inference that dynamically switches between different-sized models based on prediction uncertainty. By monitoring rolling entropy in model logit distributions, our method identifies text regions where a smaller model suffices and switches to a larger model only when prediction uncertainty exceeds a threshold. Unlike speculative decoding approaches that maintain perfect output fidelity through verification, EAD accepts controlled output divergence in exchange for computational efficiency. Our experiments on the MATH benchmark demonstrate remarkable efficiency gains across different model families. Using the LLaMA family, we maintain 96.7% of the 11B model's performance (50.4% vs 52.1%) while using it for only 43% of tokens, decreasing computational cost by 41.5%. These gains become more pronounced with larger size differentials in the Qwen family, where we achieve 92.9% of the 14B model's performance (74.3% vs 80.0%) while using it for just 25% of tokens, decreasing computational cost by 67%. The consistency of these results across model pairs suggests that language model computation can be significantly optimized by selectively deploying model capacity based on local generation complexity. Our findings indicate that current approaches to model inference may be unnecessarily conservative in their pursuit of perfect output fidelity, and that accepting minor performance trade-offs can enable dramatic reductions in computational costs.