# ShiftAddLLM: Accelerating Pretrained LLMs via Post-Training Multiplication-Less Reparameterization

arXiv: https://arxiv.org/abs/2406.05981 · HTML: https://mlsanity.com/paper/2406.05981/

## Key facts

- Cited by 30 papers in the map (data through 14 Oct 2026).
- Builds on 48 of its 74 references that are in the map.
- First posted on arXiv 10 Jun 2024.

## Builds on (most cited first, 30 of 48)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness](https://mlsanity.com/paper/2205.14135/) (2022)
- [OPT: Open Pre-trained Transformer Language Models](https://mlsanity.com/paper/2205.01068/) (2022)
- [Mistral 7B](https://mlsanity.com/paper/2310.06825/) (2023)
- [Pointer Sentinel Mixture Models](https://mlsanity.com/paper/1609.07843/) (2016)
- [BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions](https://mlsanity.com/paper/1905.10044/) (2019)
- [BLOOM: A 176B-Parameter Open-Access Multilingual Language Model](https://mlsanity.com/paper/2211.05100/) (2022)
- [SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models](https://mlsanity.com/paper/2211.10438/) (2022)
- [A Simple and Effective Pruning Approach for Large Language Models](https://mlsanity.com/paper/2306.11695/) (2023)
- [LLM-Pruner: On the Structural Pruning of Large Language Models](https://mlsanity.com/paper/2305.11627/) (2023)
- [ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformers](https://mlsanity.com/paper/2206.01861/) (2022)
- [Gated Linear Attention Transformers with Hardware-Efficient Training](https://mlsanity.com/paper/2312.06635/) (2023)
- [OmniQuant: Omnidirectionally Calibrated Quantization for Large Language Models](https://mlsanity.com/paper/2308.13137/) (2023)
- [QuIP: 2-Bit Quantization of Large Language Models With Guarantees](https://mlsanity.com/paper/2307.13304/) (2023)
- [LLM-QAT: Data-Free Quantization Aware Training for Large Language Models](https://mlsanity.com/paper/2305.17888/) (2023)
- [Optimal Brain Compression: A Framework for Accurate Post-Training Quantization and Pruning](https://mlsanity.com/paper/2208.11580/) (2022)
- [The case for 4-bit precision: k-bit Inference Scaling Laws](https://mlsanity.com/paper/2212.09720/) (2022)
- [The Unreasonable Ineffectiveness of the Deeper Layers](https://mlsanity.com/paper/2403.17887/) (2024)
- [LLM Inference Unveiled: Survey and Roofline Model Insights](https://mlsanity.com/paper/2402.16363/) (2024)
- [Additive Powers-of-Two Quantization: An Efficient Non-uniform Discretization for Neural Networks](https://mlsanity.com/paper/1909.13144/) (2019)
- [BiLLM: Pushing the Limit of Post-Training Quantization for LLMs](https://mlsanity.com/paper/2402.04291/) (2024)
- [LUT-GEMM: Quantized Matrix Multiplication based on LUTs for Efficient Inference in Large-Scale Generative Language Models](https://mlsanity.com/paper/2206.09557/) (2022)
- [Shift: A Zero FLOP, Zero Parameter Alternative to Spatial Convolutions](https://mlsanity.com/paper/1711.08141/) (2017)
- [AI and Memory Wall](https://mlsanity.com/paper/2403.14123/) (2024)
- [AdderNet: Do We Really Need Multiplications in Deep Learning?](https://mlsanity.com/paper/1912.13200/) (2019)
- [A Systematic Classification of Knowledge, Reasoning, and Context within the ARC Dataset](https://mlsanity.com/paper/1806.00358/) (2018)
- [DeepShift: Towards Multiplication-Less Neural Networks](https://mlsanity.com/paper/1905.13298/) (2019)

## Built on by (most cited first, 30 of 30)

- [MicroScopiQ: Accelerating Foundational Models through Outlier-Aware Microscaling Quantization](https://mlsanity.com/paper/2411.05282/) (2024)
- [Mitigating Hallucinations in Vision-Language Models through Image-Guided Head Suppression](https://mlsanity.com/paper/2505.16411/) (2025)
- [FIGLUT: An Energy-Efficient Accelerator Design for FP-INT GEMM Using Look-Up Tables](https://mlsanity.com/paper/2503.06862/) (2025)
- [AnyBCQ: Hardware Efficient Flexible Binary-Coded Quantization for Multi-Precision LLMs](https://mlsanity.com/paper/2510.10467/) (2025)
- [Neuromorphic Principles for Efficient Large Language Models on Intel Loihi 2](https://mlsanity.com/paper/2503.18002/) (2025)
- [TerEffic: Highly Efficient Ternary LLM Inference on FPGA](https://mlsanity.com/paper/2502.16473/) (2025)
- [DMQ: Dissecting Outliers of Diffusion Models for Post-Training Quantization](https://mlsanity.com/paper/2507.12933/) (2025)
- [Unifying Uniform and Binary-coding Quantization for Accurate Compression of Large Language Models](https://mlsanity.com/paper/2506.03781/) (2025)
- [LVLM-Compress-Bench: Benchmarking the Broader Impact of Large Vision-Language Model Compression](https://mlsanity.com/paper/2503.04982/) (2025)
- [CodeGEMM: A Codebook-Centric Approach to Efficient GEMM in Quantized LLMs](https://mlsanity.com/paper/2512.17970/) (2025)
- [Matmul or No Matmul in the Era of 1-bit LLMs](https://mlsanity.com/paper/2408.11939/) (2024)
- [Highly Efficient and Effective LLMs with Multi-Boolean Architectures](https://mlsanity.com/paper/2505.22811/) (2025)
- [EVA: Accelerating LLM Decoding via an Efficient Vector Quantization Architecture](https://mlsanity.com/paper/2605.24144/) (2026)
- [PIM-LLM: A High-Throughput Hybrid PIM Architecture for 1-bit LLMs](https://mlsanity.com/paper/2504.01994/) (2025)
- [QueueEDIT: Structural Self-Correction for Sequential Model Editing in LLMs](https://mlsanity.com/paper/2506.17864/) (2025)
- [IPTQ-ViT: Post-Training Quantization of Non-linear Functions for Integer-only Vision Transformers](https://mlsanity.com/paper/2511.15369/) (2025)
- [SFMP: Fine-Grained, Hardware-Friendly and Search-Free Mixed-Precision Quantization for Large Language Models](https://mlsanity.com/paper/2602.01027/) (2026)
- [PoTAcc: A Pipeline for End-to-End Acceleration of Power-of-Two Quantized DNNs](https://mlsanity.com/paper/2605.06082/) (2026)
- [GoQuant: Geometric Orthogonal Residual Projection for Multiplier-Free Power-of-Two Transformer Quantization](https://mlsanity.com/paper/2605.26092/) (2026)
- [A Motion-Aware Vector Quantization Framework with Centroid Reuse for Efficient VLA Inference](https://mlsanity.com/paper/2607.24148/) (2026)
- [AxLLM: accelerator architecture for large language models with computation reuse capability](https://mlsanity.com/paper/2509.22512/) (2025)
- [Binary Quadratic Quantization: Beyond First-Order Quantization for Real-Valued Matrix Compression](https://mlsanity.com/paper/2510.18650/) (2025)
- [ELUTQ: Optimizing Quantization Accuracy under LUT-Based Computation for Edge LLMs](https://mlsanity.com/paper/2510.19482/) (2025)
- [ESACT: An End-to-End Sparse Accelerator for Compute-Intensive Transformers via Local Similarity](https://mlsanity.com/paper/2512.02403/) (2025)
- [SignRoundV2: Toward Closing the Performance Gap in Extremely Low-Bit Post-Training Quantization for LLMs](https://mlsanity.com/paper/2512.04746/) (2025)
- [M2K: Making the Model-Kernel Interface Explicit for Reliable CUDA Kernel Verification](https://mlsanity.com/paper/2603.24595/) (2026)
- [QFlash: Bridging Quantization and Memory Efficiency in Vision Transformer Attention](https://mlsanity.com/paper/2604.25306/) (2026)
- [Fast-TurboQuant: A Multiplier-Free Online Vector Quantization Approach](https://mlsanity.com/paper/2606.21448/) (2026)
- [DeVIT: Low-Power Vision Transformer Acceleration Using Delta Computation](https://mlsanity.com/paper/2608.01343/) (2026)
- [QTEA: Ternary LLMs with Sparse Residual Salient Weight and By-Column Optimization](https://mlsanity.com/paper/2609.00224/) (2026)

## Abstract (arXiv)

Large language models (LLMs) have shown impressive performance on language tasks but face challenges when deployed on resource-constrained devices due to their extensive parameters and reliance on dense multiplications, resulting in high memory demands and latency bottlenecks. Shift-and-add reparameterization offers a promising solution by replacing costly multiplications with hardware-friendly primitives in both the attention and multi-layer perceptron (MLP) layers of an LLM. However, current reparameterization techniques require training from scratch or full parameter fine-tuning to restore accuracy, which is resource-intensive for LLMs. To address this, we propose accelerating pretrained LLMs through post-training shift-and-add reparameterization, creating efficient multiplication-free models, dubbed ShiftAddLLM. Specifically, we quantize each weight matrix into binary matrices paired with group-wise scaling factors. The associated multiplications are reparameterized into (1) shifts between activations and scaling factors and (2) queries and adds according to the binary matrices. To reduce accuracy loss, we present a multi-objective optimization method to minimize both weight and output activation reparameterization errors. Additionally, based on varying sensitivity across layers to reparameterization, we develop an automated bit allocation strategy to further reduce memory usage and latency. Experiments on five LLM families and eight tasks consistently validate the effectiveness of ShiftAddLLM, achieving average perplexity improvements of 5.6 and 22.7 points at comparable or lower latency compared to the most competitive quantized LLMs at 3 and 2 bits, respectively, and more than 80% memory and energy reductions over the original LLMs. Codes and models are available at https://github.com/GATECH-EIC/ShiftAddLLM.