mlsanityData to 7 Oct 2026

TIDE: Token-Informed Depth Execution for Per-Token Early Exit in LLM Inference

arXiv 2603.21365 ↗First posted 22 Mar 2026cs.LG cs.CLcited by 0

On the map: Efficient LLM inference and serving → Early exit and edge inference

Abstract (arXiv)

Large language models run every token through every layer, regardless of difficulty. We present TIDE, a post-training system that attaches tiny learned routers at periodic checkpoint layers and, at inference time, selects the earliest layer whose hidden state has converged for each token. TIDE requires no model retraining, works with any HuggingFace causal LM, auto-detects GPU architecture, and supports float32, float16, and bfloat16 through fused CUDA kernels. On an NVIDIA A100 with DeepSeek R1 Distill 8B, TIDE achieves 100% prefill exit rate (5% of tokens exit at layer 11, the remaining at layer 31), reduces prefill latency by 7.2%, and increases single-batch throughput by 6.6%. During autoregressive decoding, 98-99% of tokens exit early while the model correctly solves a multi-step math problem with 95 unique output tokens. On Qwen3 8B (36 layers), throughput improves by 8.1% at batch size 8. Calibration on 2,000 WikiText samples takes under 3 minutes and produces a ~4 MB router checkpoint. The system comprises 1,308 lines of Python and 1,081 lines of CUDA/C++ with 74 passing tests. Code: https://github.com/RightNow-AI/TIDE

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Early exit and edge inference, one of 12 directions in Efficient LLM inference and serving. Builds on 25 of its 30 references that are in the map. First posted on arXiv 22 Mar 2026.

Builds on

25 of 30 references.

Closest ancestors · 5

Same direction, newest first

  1. 2026ADEPT: Adaptive Dynamic Early-Exit Process for Transformers7 Jan
  2. 2024Early Exit Is a Natural Capability in Transformer-based Models: An Empirical Study on Early Exit without Joint Optimization2 Dec
  3. 2024LayerSkip: Enabling Early Exit Inference and Self-Speculative Decoding25 Apr
  4. 2023EE-LLM: Large-Scale Training and Inference of Early-Exit Large Language Models with 3D Parallelism8 Dec
  5. 2022Confident Adaptive Language Modeling14 Jul

Other references · 20

  1. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  2. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  3. 2013Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computationcited by 2,031
  4. 2022Fast Inference from Transformers via Speculative Decodingcited by 1,615
  5. 2023Accelerating Large Language Model Decoding with Speculative Samplingcited by 975
  6. 2018Universal Transformerscited by 613
  7. 2017BranchyNet: Fast Inference via Early Exiting from Deep Neural Networkscited by 591
  8. 2024EAGLE: Speculative Sampling Requires Rethinking Feature Uncertaintycited by 564
  9. 2016Adaptive Computation Time for Recurrent Neural Networkscited by 479
  10. 2021Dynamic Neural Networks: A Surveycited by 405
  11. 2020DeeBERT: Dynamic Early Exiting for Accelerating BERT Inferencecited by 322
  12. 2020BERT Loses Patience: Fast and Robust Inference with Early Exitcited by 267
8 more
  1. 2024Mixture-of-Depths: Dynamically allocating compute in transformer-based language modelscited by 242
  2. 2020FastBERT: a Self-distilling BERT with Adaptive Inference Timecited by 224
  3. 2024A Survey on Efficient Inference for Large Language Modelscited by 197
  4. 2023SkipDecode: Autoregressive Skip Decoding with Batching and Caching for Efficient LLM Inferencecited by 89
  5. 2024Layer-Condensed KV Cache for Efficient Inference of Large Language Modelscited by 58
  6. 2025SpecEE: Accelerating Large Language Model Inference with Speculative Early Exitingcited by 23
  7. 2024Accelerating Inference in Large Language Models with a Unified Layer Skipping Strategycited by 17
  8. 2024Conditional computation in neural networks: principles and research trendscited by 10

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026River-LLM: Large Language Model Seamless Exit Based on KV Share
  2. 2026TIDE: Temporal Incremental Draft Engine for Self-Improving LLM Inference
  3. 2025QuickSilver -- Speeding up LLM Inference through Dynamic Token Halting, KV Skipping, Contextual Token Fusion, and Adaptive Matryoshka Quantization
  4. 2025HELIOS: Adaptive Model And Early-Exit Selection for Efficient LLM Inference Serving
  5. 2025Efficient LLM Inference with Activation Checkpointing and Hybrid Caching
  6. 2023SARATHI: Efficient LLM Inference by Piggybacking Decodes with Chunked Prefills
  7. 2026FlexEE: Self-Speculative and KV-Compatible Early Exiting for Offloading-Aware LLM Inference
  8. 2026DualDecoder: Accelerate Long Context LLM Inference by Predictive Prefetch