DECA: A Near-Core LLM Decompression Accelerator Grounded on a 3D Roofline Model
Not on the map. Why?
Abstract (arXiv)
To alleviate the memory bandwidth bottleneck in Large Language Model (LLM) inference workloads, weight matrices are stored in memory in quantized and sparsified formats. Hence, before tiles of these matrices can be processed by in-core generalized matrix multiplication (GeMM) hardware engines, they need to be dequantized and de-sparsified. This is currently performed in software with vector operations. Unfortunately, this approach delivers only modest performance. Moreover, it is hard to understand how to improve the system, as the overall GeMM performance depends on the interaction between memory resources, vector units, and hardware matrix engines. To improve the performance of LLM inference in advanced platforms equipped with in-core GeMM engines and HBM, this paper makes three main contributions. First, it develops an analytical performance model with a 3D visual representation that provides insights into how memory resources, vector units, and hardware matrix engines interact to deliver compressed GeMM performance. Second, it proposes DECA, a new near-core ML-model decompression accelerator. DECA offloads tile de-sparsification and dequantization from the CPU, producing ready-to-use tiles for in-core GeMM engines. Third, it introduces a new ISA extension that enables out-of-order invocation of the near-core accelerator. With this extension, accelerator and core computations can interleave and overlap with high-performance. Our evaluation shows that, in a simulated 56-core Xeon 4 server with HBM, DECA accelerates the execution of compressed GeMMs by up to 4x over the use of optimized Intel software kernels. Further, DECA reduces the next-token generation time of Llama2-70B and OPT-66B by 1.6x-2.6x.
Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 32 of its 88 references that are in the map. First posted on arXiv 25 May 2025.
Builds on
32 of 88 references.
References · 32
- 2017Attention Is All You Needcited by 60,361
- 2015Deep Residual Learning for Image Recognitioncited by 52,096
- 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
- 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
- 2015Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Codingcited by 2,329
- 2023SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shotcited by 1,121
- 2020What is the State of Neural Network Pruning?cited by 583
- 2021Sparsity in Deep Learning: Pruning and growth for efficient inference and training in neural networkscited by 527
- 2016EIE: Efficient Inference Engine on Compressed Deep Neural Networkcited by 295
- 2023Atom: Low-bit Quantization for Efficient and Accurate LLM Servingcited by 272
- 2021Pruning and Quantization for Deep Neural Network Acceleration: A Surveycited by 263
- 2020SpAtten: Efficient Sparse Attention Architecture with Cascade Token and Head Pruningcited by 234
20 more
- 2023TPU v4: An Optically Reconfigurable Supercomputer for Machine Learning with Hardware Support for Embeddingscited by 202
- 2017SCNN: An Accelerator for Compressed-sparse Convolutional Neural Networkscited by 135
- 2021Group Fisher Pruning for Practical Network Compressioncited by 131
- 2020A3: Accelerating Attention Mechanisms in Neural Networks with Approximationcited by 101
- 2023Memory-Efficient Fine-Tuning of Compressed Large Language Models via sub-4-bit Integer Quantizationcited by 100
- 2018NVIDIA Tensor Core Programmability, Performance & Precisioncited by 79
- 2024Towards Greener LLMs: Bringing Energy-Efficiency to the Forefront of LLM Inferencecited by 61
- 2021AC/DC: Alternating Compressed/DeCompressed Training of Deep Neural Networkscited by 59
- 2023SummIt: Iterative Text Summarization via ChatGPTcited by 54
- 2024Benchmarking and Dissecting the Nvidia Hopper GPU Architecturecited by 24
- 2024Effective Interplay between Sparsity and Quantization: From Theory to Practicecited by 19
- 2023Flash-LLM: Enabling Cost-Effective and Highly-Efficient Large Generative Model Inference with Unstructured Sparsitycited by 17
- 2018The ARM Scalable Vector Extensioncited by 9
- 2021Tensor Processing Primitives: A Programming Abstraction for Efficiency and Portability in Deep Learning & HPC Workloadscited by 9
- 2023VEGETA: Vertically-Integrated Extensions for Sparse/Dense GEMM Tile Acceleration on CPUscited by 9
- 2024HLAT: High-quality Large Language Model Pre-trained on AWS Trainiumcited by 8
- 2023Harnessing Deep Learning and HPC Kernels via High-Level Loop and Tensor Abstractions on CPU Architecturescited by 5
- 2023MEGA: A Memory-Efficient GNN Accelerator Exploiting Degree-Aware Mixed-Precision Quantizationcited by 4
- 2022ReGraph: Scaling Graph Processing on HBM-enabled FPGAs with Heterogeneous Pipelinescited by 3
- 2024DeMM: A Decoupled Matrix Multiplication Engine Supporting Relaxed Structured Sparsitycited by 3
Built on by
Citing papers per year: 2026: 2.
2 papers.
Similar, not cited
Close in wording, no citation either way
- 2024Communication Compression for Tensor Parallel LLM Inference
- 2023Efficient LLM inference solution on Intel GPU
- 2025Hybrid Systolic Array Accelerator with Optimized Dataflow for Edge Large Language Model Inference
- 2025MCBP: A Memory-Compute Efficient LLM Inference Accelerator Leveraging Bit-Slice-enabled Sparsity and Repetitiveness
- 2024Anda: Unlocking Efficient LLM Inference with a Variable-Length Grouped Activation Data Format
- 2025Reimagining Memory Access for LLM Inference: Compression-Aware Memory Controller Design
- 2025SAIL: SRAM-Accelerated LLM Inference System with Lookup-Table-based GEMV
- 2024LUT Tensor Core: A Software-Hardware Co-Design for LUT-Based Low-Bit LLM Inference