mlsanityData to 14 Oct 2026

RWKV-edge: Deeply Compressed RWKV for Resource-Constrained Devices

arXiv 2412.10856 ↗First posted 14 Dec 2024cs.LG cs.PFcited by 1

Not on the map. Why?

Abstract (arXiv)

To deploy LLMs on resource-contained platforms such as mobile robots and smartphones, non-transformers LLMs have achieved major breakthroughs. Recently, a novel RNN-based LLM family, Repentance Weighted Key Value (RWKV) has shown strong computational efficiency; nevertheless, RWKV models still have high parameter counts which limited their deployment. In this paper, we propose a suite of compression techniques, ranging from model architecture optimizations to post-training compression, tailored to the RWKV architecture. Combined, our techniques reduce the memory footprint of RWKV models by 3.4x -- 5x with only negligible degradation in accuracy; compared to transformer LLMs with similar accuracy, our models require 4x less memory footprint.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 23 of its 30 references that are in the map. First posted on arXiv 14 Dec 2024.

Builds on

23 of 30 references.

References · 23

  1. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  2. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,411
  3. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,262
  4. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,919
  5. 2016The LAMBADA dataset: Word prediction requiring a broad discourse contextcited by 992
  6. 2023RWKV: Reinventing RNNs for the Transformer Eracited by 827
  7. 2018Fundamentals of Recurrent Neural Network (RNN) and Long Short-Term Memory (LSTM) Networkcited by 420
  8. 2023SpecInfer: Accelerating Generative Large Language Model Serving with Tree-based Speculative Inference and Verificationcited by 365
  9. 2023Deja Vu: Contextual Sparsity for Efficient LLMs at Inference Timecited by 252
  10. 2022The case for 4-bit precision: k-bit Inference Scaling Lawscited by 251
  11. 2022Language model compression with weighted low-rank factorizationcited by 195
  12. 2016Exploring the Limits of Language Modelingcited by 173
11 more
  1. 2024Eagle and Finch: RWKV with Matrix-Valued States and Dynamic Recurrencecited by 159
  2. 2024PowerInfer-2: Fast Large Language Model Inference on a Smartphonecited by 103
  3. 2024RAGCache: Efficient Knowledge Caching for Retrieval-Augmented Generationcited by 98
  4. 2023Large Language Models Enable Few-Shot Clusteringcited by 72
  5. 2023Meta-in-context learning in large language modelscited by 59
  6. 2024ProSparse: Introducing and Enhancing Intrinsic Activation Sparsity within Large Language Modelscited by 45
  7. 2024Turbo Sparse: Achieving LLM SOTA Performance with Minimal Activated Parameterscited by 38
  8. 2018GroupReduce: Block-Wise Low-Rank Approximation for Neural Language Model Shrinkingcited by 32
  9. 2018Online Embedding Compression for Text Classification using Low Rank Matrix Factorizationcited by 22
  10. 2024CHAI: Clustered Head Attention for Efficient LLM Inferencecited by 13
  11. 2024Context-Aware Clustering using Large Language Modelscited by 9

Built on by

Cited by year

Citing papers per year: 2024: 1.

1 paper.

  1. 2024A Survey of RWKVcited by 6