# BitMoD: Bit-serial Mixture-of-Datatype LLM Acceleration arXiv: https://arxiv.org/abs/2411.11745 · HTML: https://mlsanity.com/paper/2411.11745/ ## Key facts - Cited by 18 papers in the map (data through 14 Oct 2026). - Builds on 36 of its 63 references that are in the map. - First posted on arXiv 18 Nov 2024. ## Builds on (most cited first, 30 of 36) - [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020) - [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023) - [Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge](https://mlsanity.com/paper/1803.05457/) (2018) - [HellaSwag: Can a Machine Really Finish Your Sentence?](https://mlsanity.com/paper/1905.07830/) (2019) - [OPT: Open Pre-trained Transformer Language Models](https://mlsanity.com/paper/2205.01068/) (2022) - [QLoRA: Efficient Finetuning of Quantized LLMs](https://mlsanity.com/paper/2305.14314/) (2023) - [Pointer Sentinel Mixture Models](https://mlsanity.com/paper/1609.07843/) (2016) - [PIQA: Reasoning about Physical Commonsense in Natural Language](https://mlsanity.com/paper/1911.11641/) (2019) - [SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models](https://mlsanity.com/paper/2211.10438/) (2022) - [AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration](https://mlsanity.com/paper/2306.00978/) (2023) - [LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale](https://mlsanity.com/paper/2208.07339/) (2022) - [QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMs](https://mlsanity.com/paper/2404.00456/) (2024) - [FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPU](https://mlsanity.com/paper/2303.06865/) (2023) - [ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformers](https://mlsanity.com/paper/2206.01861/) (2022) - [Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpus](https://mlsanity.com/paper/2104.08758/) (2021) - [OmniQuant: Omnidirectionally Calibrated Quantization for Large Language Models](https://mlsanity.com/paper/2308.13137/) (2023) - [QuIP: 2-Bit Quantization of Large Language Models With Guarantees](https://mlsanity.com/paper/2307.13304/) (2023) - [8-bit Optimizers via Block-wise Quantization](https://mlsanity.com/paper/2110.02861/) (2021) - [LLM-QAT: Data-Free Quantization Aware Training for Large Language Models](https://mlsanity.com/paper/2305.17888/) (2023) - [Atom: Low-bit Quantization for Efficient and Accurate LLM Serving](https://mlsanity.com/paper/2310.19102/) (2023) - [EfficientQAT: Efficient Quantization-Aware Training for Large Language Models](https://mlsanity.com/paper/2407.11062/) (2024) - [GOBO: Quantizing Attention-Based NLP Models for Low Latency and Energy Efficient Inference](https://mlsanity.com/paper/2005.03842/) (2020) - [OliVe: Accelerating Large Language Models via Hardware-friendly Outlier-Victim Pair Quantization](https://mlsanity.com/paper/2304.07493/) (2023) - [AI and Memory Wall](https://mlsanity.com/paper/2403.14123/) (2024) - [VS-Quant: Per-vector Scaled Quantization for Accurate Low-Precision Neural Network Inference](https://mlsanity.com/paper/2102.04503/) (2021) - [ANT: Exploiting Adaptive Numerical Data Type for Low-bit Deep Neural Network Quantization](https://mlsanity.com/paper/2208.14286/) (2022) - [With Shared Microexponents, A Little Shifting Goes a Long Way](https://mlsanity.com/paper/2302.08007/) (2023) - [Bit-pragmatic Deep Neural Network Computing](https://mlsanity.com/paper/1610.06920/) (2016) - [Dual-side Sparse Tensor Core](https://mlsanity.com/paper/2105.09564/) (2021) - [Mokey: Enabling Narrow Fixed-Point Inference for Out-of-the-Box Floating-Point Transformer Models](https://mlsanity.com/paper/2203.12758/) (2022) ## Built on by (most cited first, 18 of 18) - [Is Finer Better? The Limits of Microscaling Formats in Large Language Models](https://mlsanity.com/paper/2601.19026/) (2026) - [M2XFP: A Metadata-Augmented Microscaling Data Format for Efficient Low-bit Quantization](https://mlsanity.com/paper/2601.19213/) (2026) - [FlexiBit: Fully Flexible Precision Bit-parallel Accelerator Architecture for Arbitrary Mixed Precision AI](https://mlsanity.com/paper/2411.18065/) (2024) - [Mixed-Precision Quantization for Language Models: Techniques and Prospects](https://mlsanity.com/paper/2510.16805/) (2025) - [V-Rex: Real-Time Streaming Video LLM Acceleration via Dynamic KV Cache Retrieval](https://mlsanity.com/paper/2512.12284/) (2025) - [CARVQ: Corrective Adaptor with Group Residual Vector Quantization for LLM Embedding Compression](https://mlsanity.com/paper/2510.12721/) (2025) - [P3-LLM: An Integrated NPU-PIM Accelerator for Edge LLM Inference Using Hybrid Numerical Formats](https://mlsanity.com/paper/2511.06838/) (2025) - [Focus: A Streaming Concentration Architecture for Efficient Vision-Language Models](https://mlsanity.com/paper/2512.14661/) (2025) - [A Spatio-Temporal Expert Prefetching Framework for Efficient MoE-based LLM Inference](https://mlsanity.com/paper/2606.15453/) (2026) - [HybridEP: Scaling Expert Parallelism to Cross-Datacenter Scenario via Hybrid Expert/Data Transmission](https://mlsanity.com/paper/2510.19470/) (2025) - [Platinum: Path-Adaptable LUT-Based Accelerator Tailored for Low-Bit Weight Matrix Multiplication](https://mlsanity.com/paper/2511.21910/) (2025) - [ITERA-LLM: Boosting Sub-8-Bit Large Language Model Inference via Iterative Tensor Decomposition](https://mlsanity.com/paper/2505.08981/) (2025) - [VersaQ-3D: Architecture Support for Visual Geometry Grounded Transformers via Versatile Quantization](https://mlsanity.com/paper/2601.20317/) (2026) - [MX-SAFE: Versatile Inference- and Training-Proof Microscaling Format with On-the-Fly Exponent and Mantissa Bit Allocation](https://mlsanity.com/paper/2605.24391/) (2026) - [GyRot: Leveraging Hidden Synergy between Rotation and Fine-grained Group Quantization for Low-bit LLM Inference](https://mlsanity.com/paper/2607.27694/) (2026) - [FlexPosit: Tunable Fractional Precision for LLM Inference Accelerators](https://mlsanity.com/paper/2609.04724/) (2026) - [MiX: Micro-Inverted-Scaling for End-to-End Low-Bit Vision-Language Model Acceleration](https://mlsanity.com/paper/2609.19683/) (2026) - [Precision As You Need: Stochastic Computing Is a Dense Adaptive Quantizer](https://mlsanity.com/paper/2609.32922/) (2026) ## Abstract (arXiv) Large language models (LLMs) have demonstrated remarkable performance across various machine learning tasks. Yet the substantial memory footprint of LLMs significantly hinders their deployment. In this paper, we improve the accessibility of LLMs through BitMoD, an algorithm-hardware co-design solution that enables efficient LLM acceleration at low weight precision. On the algorithm side, BitMoD introduces fine-grained data type adaptation that uses a different numerical data type to quantize a group of (e.g., 128) weights. Through the careful design of these new data types, BitMoD is able to quantize LLM weights to very low precision (e.g., 4 bits and 3 bits) while maintaining high accuracy. On the hardware side, BitMoD employs a bit-serial processing element to easily support multiple numerical precisions and data types; our hardware design includes two key innovations: First, it employs a unified representation to process different weight data types, thus reducing the hardware cost. Second, it adopts a bit-serial dequantization unit to rescale the per-group partial sum with minimal hardware overhead. Our evaluation on six representative LLMs demonstrates that BitMoD significantly outperforms state-of-the-art LLM quantization and acceleration methods. For discriminative tasks, BitMoD can quantize LLM weights to 4-bit with