# MVDRAM: Enabling GeMV Execution in Unmodified DRAM for Low-Bit LLM Acceleration

arXiv: https://arxiv.org/abs/2503.23817 · HTML: https://mlsanity.com/paper/2503.23817/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 29 of its 60 references that are in the map.
- First posted on arXiv 31 Mar 2025.

## Builds on (most cited first, 29 of 29)

- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Phi-4 Technical Report](https://mlsanity.com/paper/2412.08905/) (2024)
- [QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMs](https://mlsanity.com/paper/2404.00456/) (2024)
- [QuIP: 2-Bit Quantization of Large Language Models With Guarantees](https://mlsanity.com/paper/2307.13304/) (2023)
- [QuIP#: Even Better LLM Quantization with Hadamard Incoherence and Lattice Codebooks](https://mlsanity.com/paper/2402.04396/) (2024)
- [Atom: Low-bit Quantization for Efficient and Accurate LLM Serving](https://mlsanity.com/paper/2310.19102/) (2023)
- [QServe: W4A8KV4 Quantization and System Co-design for Efficient LLM Serving](https://mlsanity.com/paper/2405.04532/) (2024)
- [Compressed Chain of Thought: Efficient Reasoning Through Dense Representations](https://mlsanity.com/paper/2412.13171/) (2024)
- [BitNet: Scaling 1-bit Transformers for Large Language Models](https://mlsanity.com/paper/2310.11453/) (2023)
- [SeerAttention: Learning Intrinsic Sparse Attention in Your LLMs](https://mlsanity.com/paper/2410.13276/) (2024)
- [LUT-GEMM: Quantized Matrix Multiplication based on LUTs for Efficient Inference in Large-Scale Generative Language Models](https://mlsanity.com/paper/2206.09557/) (2022)
- [BitDistiller: Unleashing the Potential of Sub-4-Bit LLMs via Self-Distillation](https://mlsanity.com/paper/2402.10631/) (2024)
- [OneBit: Towards Extremely Low-bit Large Language Models](https://mlsanity.com/paper/2402.11295/) (2024)
- [VPTQ: Extreme Low-bit Vector Post-Training Quantization for Large Language Models](https://mlsanity.com/paper/2409.17066/) (2024)
- [T-MAC: CPU Renaissance via Table Lookup for Low-Bit LLM Deployment on Edge](https://mlsanity.com/paper/2407.00088/) (2024)
- [Training-Free Activation Sparsity in Large Language Models](https://mlsanity.com/paper/2408.14690/) (2024)
- [NeuPIMs: NPU-PIM Heterogeneous Acceleration for Batched LLM Inferencing](https://mlsanity.com/paper/2403.00579/) (2024)
- [Q-Sparse: All Large Language Models can be Fully Sparsely-Activated](https://mlsanity.com/paper/2407.10969/) (2024)
- [PAPI: Exploiting Dynamic Parallelism in Large Language Model Decoding with a Processing-In-Memory-Enabled Computing System](https://mlsanity.com/paper/2502.15470/) (2025)
- [IANUS: Integrated Accelerator based on NPU-PIM Unified Memory System](https://mlsanity.com/paper/2410.15008/) (2024)
- [QUAC-TRNG: High-Throughput True Random Number Generation Using Quadruple Row Activation in Commodity DRAM Chips](https://mlsanity.com/paper/2105.08955/) (2021)
- [LO-BCQ: Block Clustered Quantization for 4-bit (W4A4) LLM Inference](https://mlsanity.com/paper/2502.05376/) (2025)
- [pLUTo: Enabling Massively Parallel Computation in DRAM via Lookup Tables](https://mlsanity.com/paper/2104.07699/) (2021)
- [PIMphony: Overcoming Bandwidth and Capacity Inefficiency in PIM-based Long-Context LLM Inference System](https://mlsanity.com/paper/2412.20166/) (2024)
- [DRAM Bender: An Extensible and Versatile FPGA-based Infrastructure to Easily Test State-of-the-art DRAM Chips](https://mlsanity.com/paper/2211.05838/) (2022)
- [Functionally-Complete Boolean Logic in Real DRAM Chips: Experimental Characterization and Analysis](https://mlsanity.com/paper/2402.18736/) (2024)
- [Simultaneous Many-Row Activation in Off-the-Shelf DRAM Chips: Experimental Characterization and Analysis](https://mlsanity.com/paper/2405.06081/) (2024)
- [Memory-Centric Computing: Recent Advances in Processing-in-DRAM](https://mlsanity.com/paper/2412.19275/) (2024)
- [DRAMScope: Uncovering DRAM Microarchitecture and Characteristics by Issuing Memory Commands](https://mlsanity.com/paper/2405.02499/) (2024)

## Abstract (arXiv)

General matrix-vector multiplication (GeMV) remains a critical latency bottleneck in large language model (LLM) inference, even with quantized low-bit models. Processing-Using-DRAM (PUD), an analog in-DRAM computing technique, has the potential to repurpose on-device DRAM as a GeMV engine, offering additional high-throughput processing capabilities to widespread consumer devices without DRAM modifications. However, applying PUD to GeMV operations in the LLM inference pipeline incurs significant overheads textitbefore and textitafter in-DRAM computation, diminishing the benefits of its high-throughput processing capabilities. This paper presents MVDRAM, the first practical system to accelerate GeMV operations for low-bit LLM inference using unmodified DRAM. By leveraging the data sharing patterns and mathematical linearity in GeMV operations, MVDRAM orchestrates the processor and DRAM to eliminate the costs associated with pre-arranging inputs and bit-transposition of outputs required in conventional PUD approaches. Our experimental evaluation with four DDR4 DRAM modules shows that MVDRAM achieves comparable or even better inference speed than the processor-based implementation for GeMV operations in low-bit (under 4-bit) LLM. In particular, MVDRAM achieves up to 7.29× speedup and 30.5× energy efficiency for low-bit GeMV operations. For end-to-end LLM inference, MVDRAM achieves 2.18× and 1.31× throughput improvements, along with 3.04× and 2.35× energy efficiency, for 2-bit and 4-bit quantized low-bit models, respectively. MVDRAM has the potential to redefine the AI hardware landscape by demonstrating the feasibility of standard DRAM as an LLM accelerator.