mlsanityData to 7 Oct 2026

LiteStage: Latency-aware Layer Skipping for Multi-stage Reasoning

arXiv 2510.14211 ↗First posted 16 Oct 2025cs.CL cs.AIcited by 0

On the map: LLM reasoning → Test-time compute and efficient thinking

Abstract (arXiv)

Multi-stage reasoning has emerged as an effective strategy for enhancing the reasoning capability of small language models by decomposing complex problems into sequential sub-stages. However, this comes at the cost of increased latency. We observe that existing adaptive acceleration techniques, such as layer skipping, struggle to balance efficiency and accuracy in this setting due to two key challenges: (1) stage-wise variation in skip sensitivity, and (2) the generation of redundant output tokens. To address these, we propose LiteStage, a latency-aware layer skipping framework for multi-stage reasoning. LiteStage combines a stage-wise offline search that allocates optimal layer budgets with an online confidence-based generation early exit to suppress unnecessary decoding. Experiments on three benchmarks, e.g., OBQA, CSQA, and StrategyQA, show that LiteStage outperforms prior training-free layer skipping methods.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Test-time compute and efficient thinking, one of 17 directions in LLM reasoning. Builds on 24 of its 40 references that are in the map. First posted on arXiv 16 Oct 2025.

Builds on

24 of 40 references.

Closest ancestors · 5

Same direction, newest first

  1. 2025Entropy After </Think> for reasoning model early exiting30 Sep
  2. 2025Dynamic Early Exit in Reasoning Models22 Apr
  3. 2025Reasoning Models Know When They're Right: Probing Hidden States for Self-Verification7 Apr
  4. 2024Do NOT Think That Much for 2+3=? On the Overthinking of o1-Like LLMs30 Dec
  5. 2024TinyThinker: Distilling Reasoning through Coarse-to-Fine Knowledge Internalization with Self-Reflection11 Dec

Other references · 19

  1. 2025Qwen3 Technical Reportcited by 8,195
  2. 2024Qwen2.5 Technical Reportcited by 4,713
  3. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 1,906
  4. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,793
  5. 2024LLaVA-CoT: Let Vision Language Models Reason Step-by-Stepcited by 446
  6. 2024ShortGPT: Layers in Large Language Models are More Redundant Than You Expectcited by 369
  7. 2020DeeBERT: Dynamic Early Exiting for Accelerating BERT Inferencecited by 322
  8. 2024LayerSkip: Enabling Early Exit Inference and Self-Speculative Decodingcited by 302
  9. 2024Mixture-of-Depths: Dynamically allocating compute in transformer-based language modelscited by 242
  10. 2024The Impact of Reasoning Step Length on Large Language Modelscited by 152
  11. 2025Mixture-of-Recursions: Learning Dynamic Recursive Depths for Adaptive Token-Level Computationcited by 120
  12. 2024Not All Layers of LLMs Are Necessary During Inferencecited by 95
7 more
  1. 2023SkipDecode: Autoregressive Skip Decoding with Batching and Caching for Efficient LLM Inferencecited by 89
  2. 2023EE-LLM: Large-Scale Training and Inference of Early-Exit Large Language Models with 3D Parallelismcited by 69
  3. 2025Adaptive Layer-skipping in Pre-trained LLMscited by 24
  4. 2025Early Stopping Chain-of-thoughts in Large Language Modelscited by 19
  5. 2024Accelerating Inference in Large Language Models with a Unified Layer Skipping Strategycited by 17
  6. 2024Improve Student's Reasoning Generalizability through Cascading Decomposed CoTs Distillationcited by 9
  7. 2024Router-Tuning: A Simple and Effective Approach for Enabling Dynamic-Depth in Transformerscited by 8

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Correct, Concise and Complete: Multi-stage Training For Adaptive Reasoning
  2. 2025When Fewer Layers Break More Chains: Layer Pruning Harms Test-Time Scaling in LLMs
  3. 2025Less is More: Improving LLM Reasoning with Minimal Test-Time Intervention
  4. 2026Skip a Layer or Loop It? Learning Program-of-Layers in LLMs
  5. 2024AQA-Bench: An Interactive Benchmark for Evaluating LLMs' Sequential Reasoning Ability
  6. 2025Prolonged Reasoning Is Not All You Need: Certainty-Based Adaptive Routing for Efficient LLM/MLLM Reasoning
  7. 2026Stabilizing Efficient Reasoning with Step-Level Advantage Selection
  8. 2025Temporal Sampling for Forgotten Reasoning in LLMs