mlsanityData to 14 Oct 2026

Efficient Large Language Model Inference with Neural Block Linearization

arXiv 2505.21077 ↗First posted 27 May 2025cs.LG cs.AIcited by 1

On the map: Efficient LLM inference and serving → Early exit and edge inference

Abstract (arXiv)

The high inference demands of transformer-based Large Language Models (LLMs) pose substantial challenges in their deployment. To this end, we introduce Neural Block Linearization (NBL), a novel framework for accelerating transformer model inference by replacing self-attention layers with linear approximations derived from Linear Minimum Mean Squared Error estimators. NBL leverages Canonical Correlation Analysis to compute a theoretical upper bound on the approximation error. Then, we use this bound as a criterion for substitution, selecting the LLM layers with the lowest linearization error. NBL can be efficiently applied to pre-trained LLMs without the need for fine-tuning. In experiments, NBL achieves notable computational speed-ups while preserving competitive accuracy on multiple reasoning benchmarks. For instance, applying NBL to 12 self-attention layers in DeepSeek-R1-Distill-Llama-8B increases the inference speed by 32% with less than 1% accuracy trade-off, making it a flexible and promising solution to improve the inference efficiency of LLMs. The implementation is available at: https://github.com/LIONS-EPFL/NBL.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Early exit and edge inference, one of 12 directions in Efficient LLM inference and serving. Builds on 58 of its 75 references that are in the map. First posted on arXiv 27 May 2025.

Builds on

58 of 75 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023ASVD: Activation-aware Singular Value Decomposition for Compressing Large Language Models10 Dec

Other references · 57

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  4. 2023GPT-4 Technical Reportcited by 18,396
  5. 2024The Llama 3 Herd of Modelscited by 14,715
  6. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  7. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  8. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  9. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  10. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  11. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  12. 2023Mistral 7Bcited by 3,164
38 more
  1. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,883
  2. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,411
  3. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,218
  4. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,183
  5. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,164
  6. 2015Learning both Weights and Connections for Efficient Neural Networkscited by 2,031
  7. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 2,022
  8. 2019Similarity of Neural Network Representations Revisitedcited by 1,755
  9. 2022Fast Inference from Transformers via Speculative Decodingcited by 1,653
  10. 2020Rethinking Attention with Performerscited by 1,601
  11. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,346
  12. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,298
  13. 2019TinyBERT: Distilling BERT for Natural Language Understandingcited by 1,167
  14. 2019Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Prunedcited by 856
  15. 2024Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Headscited by 843
  16. 2023LLM-Pruner: On the Structural Pruning of Large Language Modelscited by 650
  17. 2021Sparsity in Deep Learning: Pruning and growth for efficient inference and training in neural networkscited by 527
  18. 2023Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruningcited by 415
  19. 2024SliceGPT: Compress Large Language Models by Deleting Rows and Columnscited by 385
  20. 2024ShortGPT: Layers in Large Language Models are More Redundant Than You Expectcited by 383
  21. 2024QuIP#: Even Better LLM Quantization with Hadamard Incoherence and Lattice Codebookscited by 378
  22. 2018Efficient Attention: Attention with Linear Complexitiescited by 368
  23. 2021Random Feature Attentioncited by 354
  24. 2025EAGLE-3: Scaling up Inference Acceleration of Large Language Models via Training-Time Testcited by 331
  25. 2018Insights on representational similarity in neural networks with canonical correlationcited by 324
  26. 2024Unlocking Efficiency in Large Language Model Inference: A Comprehensive Survey of Speculative Decodingcited by 269
  27. 2020Do Wide and Deep Networks Learn the Same Things? Uncovering How Neural Network Representations Vary with Width and Depthcited by 245
  28. 2024SVD-LLM: Truncation-aware Singular Value Decomposition for Large Language Model Compressioncited by 243
  29. 2024The Unreasonable Ineffectiveness of the Deeper Layerscited by 226
  30. 2023LoftQ: LoRA-Fine-Tuning-Aware Quantization for Large Language Modelscited by 196
  31. 2022Language model compression with weighted low-rank factorizationcited by 195
  32. 2020Soft Threshold Weight Reparameterization for Learnable Sparsitycited by 151
  33. 2024Compact Language Models via Pruning and Knowledge Distillationcited by 148
  34. 2024The Mamba in the Llama: Distilling and Accelerating Hybrid Modelscited by 122
  35. 2024SLEB: Streamlining LLMs through Redundancy Verification and Elimination of Transformer Blockscited by 97
  36. 2024What Matters in Transformers? Not All Attention is Neededcited by 84
  37. 2023ZeroQuant-V2: Exploring Post-training Quantization in LLMs from Comprehensive Study to Low Rank Compensationcited by 79
  38. 2024Model Compression and Efficient Inference for Large Language Models: A Surveycited by 64

50 most cited of 57 All 57 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026RECLAIM: Can Agents Reproduce the Claims of Machine Learning Papers?cited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024A Survey on Efficient Inference for Large Language Models
  2. 2025LLM Inference Acceleration via Efficient Operation Fusion
  3. 2024Block Transformer: Global-to-Local Language Modeling for Fast Inference
  4. 2024Star Attention: Efficient LLM Inference over Long Sequences
  5. 2024Accelerating Large Language Model Inference with Self-Supervised Early Exits
  6. 2024LLM Inference Unveiled: Survey and Roofline Model Insights
  7. 2023Revisiting Block-based Quantisation: What is Important for Sub-8-bit LLM Inference?
  8. 2025Speed Always Wins: A Survey on Efficient Architectures for Large Language Models