mlsanityData to 7 Oct 2026

Not All Layers of LLMs Are Necessary During Inference

arXiv 2403.02181 ↗First posted 4 Mar 2024cs.CL cs.AI cs.LGcited by 95

Not on the map. Why?

Abstract (arXiv)

Due to the large number of parameters, the inference phase of Large Language Models (LLMs) is resource-intensive. However, not all requests posed to LLMs are equally difficult to handle. Through analysis, we show that for some tasks, LLMs can achieve results comparable to the final output at some intermediate layers. That is, not all layers of LLMs are necessary during inference. If we can predict at which layer the inferred results match the final results (produced by evaluating all layers), we could significantly reduce the inference cost. To this end, we propose a simple yet effective algorithm named AdaInfer to adaptively terminate the inference process for an input instance. AdaInfer relies on easily obtainable statistical features and classic classifiers like SVM. Experiments on well-known LLMs like the Llama2 series and OPT, show that AdaInfer can achieve an average of 17.8% pruning ratio, and up to 43% on sentiment tasks, with nearly no performance drop (<1%). Because AdaInfer does not alter LLM parameters, the LLMs incorporated with AdaInfer maintain generalizability across tasks.

Key facts. Cited by 95 papers in the map (data through 7 Oct 2026). Builds on 39 of its 66 references that are in the map. First posted on arXiv 4 Mar 2024.

Builds on

39 of 66 references.

References · 39

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  5. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  6. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  7. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  8. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  9. 2015Character-level Convolutional Networks for Text Classificationcited by 2,293
  10. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,838
  11. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,793
  12. 2023SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shotcited by 1,090
27 more
  1. 2021Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LMcited by 621
  2. 2022Mixture-of-Experts with Expert Choice Routingcited by 611
  3. 2018Rethinking the Value of Network Pruningcited by 595
  4. 2017BranchyNet: Fast Inference via Early Exiting from Deep Neural Networkscited by 591
  5. 2021Sparsity in Deep Learning: Pruning and growth for efficient inference and training in neural networkscited by 521
  6. 2023Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruningcited by 408
  7. 2021Dynamic Neural Networks: A Surveycited by 405
  8. 2023Editing Large Language Models: Problems, Methods, and Opportunitiescited by 367
  9. 2017SkipNet: Learning Dynamic Routing in Convolutional Networkscited by 328
  10. 2022Data Distributional Properties Drive Emergent In-Context Learning in Transformerscited by 303
  11. 2023Label Words are Anchors: An Information Flow Perspective for Understanding In-Context Learningcited by 235
  12. 2019Depth-Adaptive Transformercited by 227
  13. 2020FastBERT: a Self-distilling BERT with Adaptive Inference Timecited by 224
  14. 2021Efficient Large Scale Language Modeling with Mixtures of Expertscited by 175
  15. 2017Adaptive Neural Networks for Efficient Inferencecited by 161
  16. 2020Resolution Adaptive Networks for Efficient Inferencecited by 146
  17. 2024LaCo: Large Language Model Pruning via Layer Collapsecited by 124
  18. 2021The Efficiency Misnomercited by 98
  19. 2024SLEB: Streamlining LLMs through Redundancy Verification and Elimination of Transformer Blockscited by 96
  20. 2018Channel Gating Neural Networkscited by 80
  21. 2020CascadeBERT: Accelerating Inference of Pre-trained Language Models via Calibrated Complete Models Cascadecited by 47
  22. 2019Batch-Shaping for Learning Conditional Channel Gated Networkscited by 42
  23. 2021Accelerating BERT Inference for Sequence Labeling via Early-Exitcited by 29
  24. 2023FLM-101B: An Open LLM and How to Train It with 100K Budgetcited by 22
  25. 2023Learning to Skip for Language Modelingcited by 19
  26. 2023One Pass Streaming Algorithm for Super Long Token Attention Approximation in Sublinear Spacecited by 6
  27. 2023BiPFT: Binary Pre-trained Foundation Transformer with Low-rank Estimation of Binarization Residual Polynomialscited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 21; 2025: 39; 2026: 35.

95 papers.

Sort:

By area:

  1. 2025Layer by Layer: Uncovering Hidden Representations in Language Modelscited by 293
  2. 2024LISA: Layerwise Importance Sampling for Memory-Efficient Large Language Model Fine-Tuningcited by 88
  3. 2024Defending Large Language Models Against Jailbreak Attacks via Layer-specific Editingcited by 77
  4. 2024Mobile-Bench: An Evaluation Benchmark for LLM-based Mobile Agentscited by 56
  5. 2024Usable XAI: 10 Strategies Towards Exploiting Explainability in the LLM Eracited by 45
  6. 2024DynamicKV: Task-Aware Adaptive KV Cache Compression for Long Context LLMscited by 42
  7. 2025LLM Unlearning via Neural Activation Redirectioncited by 38
  8. 2024Large Language Model Inference Acceleration: A Comprehensive Hardware Perspectivecited by 36
  9. 2025Harnessing the Reasoning Economy: A Survey of Efficient Reasoning for Large Language Modelscited by 35
  10. 2024Does Representation Matter? Exploring Intermediate Layers in Large Language Modelscited by 31
  11. 2024What Happened in LLMs Layers when Trained for Fast vs. Slow Thinking: A Gradient Perspectivecited by 29
  12. 2025SpecEE: Accelerating Large Language Model Inference with Speculative Early Exitingcited by 23
38 more
  1. 2025BiggerGait: Unlocking Gait Recognition with Layer-wise Representations from Large Vision Modelscited by 18
  2. 2025Vision Function Layer in Multimodal LLMscited by 17
  3. 2024Accelerating Multimodal Large Language Models via Dynamic Visual-Token Exit and the Empirical Findingscited by 15
  4. 2025How Do LLMs Use Their Depth?cited by 15
  5. 2024Parenting: Optimizing Knowledge Selection of Retrieval-Augmented Language Models with Parameter Decoupling and Tailored Tuningcited by 14
  6. 2025Understanding Language Prior of LVLMs by Contrasting Chain-of-Embeddingcited by 13
  7. 2025AdaSkip: Adaptive Sublayer Skipping for Accelerating Long-Context LLM Inferencecited by 12
  8. 2025LayerIF: Estimating Layer Quality for Large Language Models using Influence Functionscited by 12
  9. 2024Look Within, Why LLMs Hallucinate: A Causal Perspectivecited by 11
  10. 2025SkipGPT: Dynamic Layer Pruning Reinvented with Token Awareness and Module Decouplingcited by 11
  11. 2025PUMA: Layer-Pruned Language Model for Efficient Unified Multimodal Retrieval with Modality-Adaptive Learningcited by 11
  12. 2025HBridge: H-Shape Bridging of Heterogeneous Experts for Unified Multimodal Understanding and Generationcited by 11
  13. 2024HW-GPT-Bench: Hardware-Aware Architecture Benchmark for Language Modelscited by 10
  14. 2024Understanding Reasoning in Chain-of-Thought from the Hopfieldian Viewcited by 10
  15. 2026From LLMs to LRMs: Rethinking Pruning for Reasoning-Centric Modelscited by 9
  16. 2025Fast and Cost-effective Speculative Edge-Cloud Decoding with Early Exitscited by 8
  17. 2025Learning to Skip the Middle Layers of Transformerscited by 8
  18. 2025HiProbe-VAD: Video Anomaly Detection via Hidden States Probing in Tuning-Free Multimodal LLMscited by 8
  19. 2026DySL-VLA: Efficient Vision-Language-Action Model Inference via Dynamic-Static Layer-Skipping for Robot Manipulationcited by 8
  20. 2026Garbage Attention in Large Language Models: BOS Sink Heads and Sink-aware Pruningcited by 7
  21. 2024Large Language Models Are Overparameterized Text Encoderscited by 6
  22. 2025DLP: Dynamic Layerwise Pruning in Large Language Modelscited by 6
  23. 2025VGent: Visual Grounding via Modular Design for Disentangling Reasoning and Predictioncited by 6
  24. 2024Cross-layer Attention Sharing for Pre-trained Large Language Modelscited by 5
  25. 2025Optimizing Multi-Hop Document Retrieval Through Intermediate Representationscited by 5
  26. 2026Tracing Agentic Failure from the Flow of Successcited by 5
  27. 2024Exploring Concept Depth: How Large Language Models Acquire Knowledge and Concept at Different Layers?cited by 4
  28. 2024RAEE: A Robust Retrieval-Augmented Early Exit Framework for Efficient Inferencecited by 4
  29. 2025CoCo-CoLa: Evaluating and Improving Language Adherence in Multilingual LLMscited by 4
  30. 2025Efficient Safety Alignment of Large Language Models via Preference Re-ranking and Representation-based Reward Modelingcited by 4
  31. 2025Accelerating Mobile Language Model via Speculative Decoding and NPU-Coordinated Executioncited by 4
  32. 2026Semantic Routing: Exploring Multi-Layer LLM Feature Weighting for Diffusion Transformerscited by 4
  33. 2025Targeting Alignment: Extracting Safety Classifiers of Aligned LLMscited by 3
  34. 2025DASH: Input-Aware Dynamic Layer Skipping for Efficient LLM Inference with Markov Decision Policiescited by 3
  35. 2025TRACE for Tracking the Emergence of Semantic Representations in Transformerscited by 3
  36. 2025Stop Before You Fail: Operational Capability Boundaries for Mitigating Unproductive Reasoning in Large Reasoning Modelscited by 3
  37. 2025A Comparative Analysis of Contextual Representation Flow in State-Space and Transformer Architecturescited by 3
  38. 2026From Static Inference to Dynamic Interaction: A Survey of Streaming Large Language Modelscited by 3

50 most cited of 95 All 95 (JSON)