mlsanityData to 14 Oct 2026

RWKVQuant: Quantizing the RWKV Family with Proxy Guided Hybrid of Scalar and Vector Quantization

arXiv 2505.03803 ↗First posted 2 May 2025cs.LG cs.AIcited by 4

Not on the map. Why?

Abstract (arXiv)

RWKV is a modern RNN architecture with comparable performance to Transformer, but still faces challenges when deployed to resource-constrained devices. Post Training Quantization (PTQ), which is a an essential technique to reduce model size and inference latency, has been widely used in Transformer models. However, it suffers significant degradation of performance when applied to RWKV. This paper investigates and identifies two key constraints inherent in the properties of RWKV: (1) Non-linear operators hinder the parameter-fusion of both smooth- and rotation-based quantization, introducing extra computation overhead. (2) The larger amount of uniformly distributed weights poses challenges for cluster-based quantization, leading to reduced accuracy. To this end, we propose RWKVQuant, a PTQ framework tailored for RWKV models, consisting of two novel techniques: (1) a coarse-to-fine proxy capable of adaptively selecting different quantization approaches by assessing the uniformity and identifying outliers in the weights, and (2) a codebook optimization algorithm that enhances the performance of cluster-based quantization methods for element-wise multiplication in RWKV. Experiments show that RWKVQuant can quantize RWKV-6-14B into about 3-bit with less than 1% accuracy loss and 2.14x speed up.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Builds on 25 of its 47 references that are in the map. First posted on arXiv 2 May 2025.

Builds on

25 of 47 references.

References · 25

  1. 2017Attention Is All You Needcited by 60,361
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  4. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  5. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,883
  6. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,164
  7. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,919
  8. 2017Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inferencecited by 1,593
  9. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,386
  10. 2016Semantic Understanding of Scenes through the ADE20K Datasetcited by 1,069
  11. 2023RWKV: Reinventing RNNs for the Transformer Eracited by 827
  12. 2024QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMscited by 634
13 more
  1. 2024SpinQuant: LLM quantization with learned rotationscited by 471
  2. 2023OmniQuant: Omnidirectionally Calibrated Quantization for Large Language Modelscited by 424
  3. 2024Extreme Compression of Large Language Models via Additive Quantizationcited by 232
  4. 2024Eagle and Finch: RWKV with Matrix-Valued States and Dynamic Recurrencecited by 159
  5. 2024Vision-RWKV: Efficient and Scalable Visual Perception with RWKV-Like Architecturescited by 98
  6. 2025OstQuant: Refining Large Language Model Quantization with Orthogonal and Scaling Transformations for Better Distribution Fittingcited by 97
  7. 2021QA Dataset Explosion: A Taxonomy of NLP Resources for Question Answering and Reading Comprehensioncited by 91
  8. 2024GPTVQ: The Blessing of Dimensionality for LLM Quantizationcited by 82
  9. 2024VPTQ: Extreme Low-bit Vector Post-Training Quantization for Large Language Modelscited by 57
  10. 2018A Systematic Classification of Knowledge, Reasoning, and Context within the ARC Datasetcited by 55
  11. 2023Post-Training Quantization for Re-parameterization via Coarse & Fine Weight Splittingcited by 6
  12. 2024A Survey of RWKVcited by 6
  13. 2024OnlySportsLM: Optimizing Sports-Domain Language Models with SOTA Performance under Billion Parameterscited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 3.

4 papers.

  1. 2026FocusTrack: One-Stage Focus-and-Suppress Framework for 3D Point Cloud Object Trackingcited by 7
  2. 2026TORQ: Two-Level Orthogonal Rotation for MXFP4 Quantizationcited by 3
  3. 2025RSAVQ: Riemannian Sensitivity-Aware Vector Quantization for Large Language Modelscited by 2
  4. 2026Breaking Modality Heterogeneity in Low-Bit Quantization for Large Vision-Language Modelscited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2024RepQuant: Towards Accurate Post-Training Quantization of Large Transformer Models via Scale Reparameterization
  2. 2026TR-PTQ: High-Accuracy Integer-Only Transformer Post Training Quantization via Taylor Region Reformulation
  3. 2024FrameQuant: Flexible Low-Bit Quantization for Transformers
  4. 2026HFRWKV: A High-Performance Fully On-Chip Hardware Accelerator for RWKV ↗
  5. 2024MPTQ-ViT: Mixed-Precision Post-Training Quantization for Vision Transformer
  6. 2025Efficiently Training A Flat Neural Network Before It has been Quantizated
  7. 2024DopQ-ViT: Towards Distribution-Friendly and Outlier-Aware Post-Training Quantization for Vision Transformers
  8. 2022ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformers