mlsanityData to 14 Oct 2026

MCBP: A Memory-Compute Efficient LLM Inference Accelerator Leveraging Bit-Slice-enabled Sparsity and Repetitiveness

arXiv 2509.10372 ↗First posted 12 Sep 2025cs.ARcited by 4

On the map: Efficient LLM inference and serving → Serving systems and request scheduling

Abstract (arXiv)

Large language models (LLMs) face significant inference latency due to inefficiencies in GEMM operations, weight access, and KV cache access, especially in real-time scenarios. This highlights the need for a versatile compute-memory efficient accelerator. Unfortunately, existing Transformer accelerators struggle to address both aspects simultaneously, as they focus on value-level processing, missing fine-grained opportunities to optimize computation and memory collaboratively. This paper introduces MCBP, a bit-grained compute-memory efficient algorithm-hardware co-design that leverages bit-slice (BS) enabled repetitiveness and sparsity to accelerate LLM inference. MCBP features three key innovations: 1) BS-repetitiveness-enabled computation reduction (BRCR), which eliminates redundant GEMM computations via leveraging redundancy hidden among BS vectors; 2) BS-sparsity-enabled two-state coding (BSTC), which reduces weight access via exploiting significant sparsity in high-order bit-slice weight; 3) Bit-grained progressive prediction (BGPP), which reduces KV cache access by leveraging early-termination-based bit-grained prediction. These techniques, supported by custom accelerator designs, effectively alleviate the burden in GEMM, weight access, and KV cache access. Extensive experiments on 26 benchmarks show that MCBP achieves 9.43x speed up and 31.1x higher energy efficiency than Nvidia A100 GPU. Compared to SOTA Transformer accelerators, MCBP achieves 35x, 5.2x and 3.2x energy saving than Spatten, FACT and SOFA, respectively.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Placed in Serving systems and request scheduling, one of 12 directions in Efficient LLM inference and serving. Builds on 52 of its 114 references that are in the map. First posted on arXiv 12 Sep 2025.

Builds on

52 of 114 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024Infinite-LLM: Efficient LLM Service for Long Context with DistAttention and Distributed KVCache5 Jan

Other references · 51

  1. 2017Attention Is All You Needcited by 60,361
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  3. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  4. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  5. 2021Program Synthesis with Large Language Modelscited by 3,637
  6. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  7. 2016Pointer Sentinel Mixture Modelscited by 3,096
  8. 2015Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Codingcited by 2,329
  9. 2023Code Llama: Open Foundation Models for Codecited by 2,139
  10. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,984
  11. 2017Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inferencecited by 1,593
  12. 2023A Survey on Evaluation of Large Language Modelscited by 1,242
38 more
  1. 2023Splitwise: Efficient generative LLM inference using phase splittingcited by 558
  2. 2019Q8BERT: Quantized 8Bit BERTcited by 300
  3. 2016EIE: Efficient Inference Engine on Compressed Deep Neural Networkcited by 295
  4. 2023Atom: Low-bit Quantization for Efficient and Accurate LLM Servingcited by 272
  5. 2020SpAtten: Efficient Sparse Attention Architecture with Cascade Token and Head Pruningcited by 234
  6. 2023Using an LLM to Help With Code Understandingcited by 228
  7. 2019WinoGrande: An Adversarial Winograd Schema Challenge at Scalecited by 204
  8. 2022Outlier Suppression: Pushing the Limit of Low-bit Transformer Language Modelscited by 181
  9. 2020WikiLingua: A New Benchmark Dataset for Cross-Lingual Abstractive Summarizationcited by 154
  10. 2023Accelerating LLM Inference with Staged Speculative Decodingcited by 151
  11. 2017SCNN: An Accelerator for Compressed-sparse Convolutional Neural Networkscited by 135
  12. 2017Bit Fusion: Bit-Level Dynamically Composable Architecture for Accelerating Deep Neural Networkscited by 109
  13. 2020A3: Accelerating Attention Mechanisms in Neural Networks with Approximationcited by 101
  14. 2024CodeAid: Evaluating a Classroom Deployment of an LLM-based Programming Assistant that Balances Student and Educator Needscited by 84
  15. 2020EdgeBERT: Sentence-Level Energy Optimizations for Latency-Aware Multi-Task NLP Inferencecited by 70
  16. 2020FTRANS: Energy-Efficient Acceleration of Transformers using FPGAcited by 54
  17. 2022DFX: A Low-latency Multi-FPGA Appliance for Accelerating Transformer-based Text Generationcited by 48
  18. 2022ViTCoD: Vision Transformer Acceleration via Dedicated Algorithm and Accelerator Co-Designcited by 41
  19. 2016Bit-pragmatic Deep Neural Network Computingcited by 32
  20. 2022Adaptable Butterfly Accelerator for Attention-based NNs via Hardware and Algorithm Co-designcited by 31
  21. 2022An Algorithm-Hardware Co-Optimized Framework for Accelerating N:M Sparse Transformerscited by 29
  22. 2021Energon: Towards Efficient Acceleration of Transformers Using Dynamic Sparse Attentioncited by 26
  23. 2023AccelTran: A Sparsity-Aware Accelerator for Dynamic Inference with Transformerscited by 24
  24. 2018UCNN: Exploiting Computational Reuse in Deep Neural Networks via Weight Repetitioncited by 20
  25. 2021S2TA: Exploiting Structured Sparsity for Energy-Efficient Mobile CNN Accelerationcited by 19
  26. 2020Procrustes: a Dataflow and Accelerator for Sparse Deep Neural Network Trainingcited by 18
  27. 2022Sparse Attention Acceleration with Synergistic In-Memory Pruning and On-Chip Recomputationcited by 12
  28. 2023HighLight: Efficient and Flexible DNN Acceleration with Hierarchical Structured Sparsitycited by 12
  29. 2022SALO: An Efficient Spatial Accelerator Enabling Hybrid Sparse Attention Mechanisms for Long Sequencescited by 10
  30. 2024SOFA: A Compute-Memory Optimized Sparsity Accelerator via Cross-Stage Coordinated Tilingcited by 9
  31. 2025LUT-DLA: Lookup Table as Efficient Extreme Low-Bit Deep Learning Acceleratorcited by 9
  32. 2022Accelerating Attention through Gradient-Based Learned Runtime Pruningcited by 8
  33. 2016YodaNN: An Architecture for Ultra-Low Power Binary-Weight CNN Accelerationcited by 7
  34. 2015Origami: A 803 GOp/s/W Convolutional Network Acceleratorcited by 6
  35. 2024SWAT: Scalable and Efficient Window Attention-based Transformers Acceleration on FPGAscited by 6
  36. 2024Panacea: Novel DNN Accelerator using Accuracy-Preserving Asymmetric Quantization and Energy-Saving Bit-Slice Sparsitycited by 5
  37. 2025QUAD: Quantization and Parameter-Efficient Tuning of LLM with Activation Decompositioncited by 4
  38. 2023Beyond the Hype: Assessing the Performance, Trustworthiness, and Clinical Suitability of GPT3.5cited by 2

50 most cited of 51 All 51 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 2.

4 papers.

  1. 2026LLMServingSim 2.0: A Unified Simulator for Heterogeneous and Disaggregated LLM Serving Infrastructurecited by 9
  2. 2025BitStopper: An Efficient Transformer Attention Accelerator via Stage-fusion and Early Terminationcited by 1
  3. 2025LAPA: Log-Domain Prediction-Driven Dynamic Sparsity Accelerator for Transformer Modelcited by 1
  4. 2026MOCAP: Wafer-Scale-Chip-Oriented Memory-Orchestrated Chunked Pipelining Framework for Prefill-Only LLM Inferencecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Reimagining Memory Access for LLM Inference: Compression-Aware Memory Controller Design
  2. 2023Efficient LLM inference solution on Intel GPU
  3. 2025Accelerating LLM Inference with Flexible N:M Sparsity via A Fully Digital Compute-in-Memory Accelerator
  4. 2026ZipServ: Fast and Memory-Efficient LLM Inference with Hardware-Aware Lossless Compression
  5. 2026LLM Inference in a Flash!
  6. 2025TENET: An Efficient Sparsity-Aware LUT-Centric Architecture for Ternary LLM Inference On Edge
  7. 2025SAIL: SRAM-Accelerated LLM Inference System with Lookup-Table-based GEMV
  8. 2023Efficient LLM Inference on CPUs