mlsanityData to 7 Oct 2026

ReALLM: A general framework for LLM compression and fine-tuning

arXiv 2405.13155 ↗First posted 21 May 2024cs.LGcited by 3

Not on the map. Why?

Abstract (arXiv)

We introduce ReALLM, a novel approach for compression and memory-efficient adaptation of pre-trained language models that encompasses most of the post-training quantization and fine-tuning methods for a budget of <4 bits. Pre-trained matrices are decomposed into a high-precision low-rank component and a vector-quantized latent representation (using an autoencoder). During the fine-tuning step, only the low-rank components are updated. Our results show that pre-trained matrices exhibit different patterns. ReALLM adapts the shape of the encoder (small/large embedding, high/low bit VQ, etc.) to each matrix. ReALLM proposes to represent each matrix with a small embedding on b bits and a neural decoder model Dφ with its weights on bφ bits. The decompression of a matrix requires only one embedding and a single forward pass with the decoder. Our weight-only quantization algorithm yields the best results on language generation tasks (C4 and WikiText-2) for a budget of 3 bits without any training. With a budget of 2 bits, ReALLM achieves state-of-the art performance after fine-tuning on a small calibration dataset.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 44 of its 53 references that are in the map. First posted on arXiv 21 May 2024.

Builds on

44 of 53 references.

References · 44

  1. 2017Attention Is All You Needcited by 59,932
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  5. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  6. 2017Neural Discrete Representation Learningcited by 4,650
  7. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  8. 2022A ConvNet for the 2020scited by 3,545
  9. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  10. 2023Mistral 7Bcited by 3,065
  11. 2016Pointer Sentinel Mixture Modelscited by 2,992
  12. 2015Learning both Weights and Connections for Efficient Neural Networkscited by 1,998
32 more
  1. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,838
  2. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,338
  3. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,234
  4. 2016Real-Time Single Image and Video Super-Resolution Using an Efficient Sub-Pixel Convolutional Neural Networkcited by 1,199
  5. 2024Gemma: Open Models Based on Gemini Research and Technologycited by 889
  6. 2021A Survey of Quantization Methods for Efficient Neural Network Inferencecited by 798
  7. 2024DoRA: Weight-Decomposed Low-Rank Adaptationcited by 740
  8. 2024QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMscited by 598
  9. 2015BinaryConnect: Training Deep Neural Networks with binary weights during propagationscited by 571
  10. 2024KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantizationcited by 571
  11. 2023The Falcon Series of Open Language Modelscited by 526
  12. 2022ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformerscited by 517
  13. 2023OmniQuant: Omnidirectionally Calibrated Quantization for Large Language Modelscited by 403
  14. 2023SpQR: A Sparse-Quantized Representation for Near-Lossless LLM Weight Compressioncited by 359
  15. 2024QuIP#: Even Better LLM Quantization with Hadamard Incoherence and Lattice Codebookscited by 354
  16. 2020Parameter-Efficient Transfer Learning with Diff Pruningcited by 351
  17. 2023SqueezeLLM: Dense-and-Sparse Quantizationcited by 285
  18. 2024Extreme Compression of Large Language Models via Additive Quantizationcited by 223
  19. 2023LoftQ: LoRA-Fine-Tuning-Aware Quantization for Large Language Modelscited by 195
  20. 2020ReActNet: Towards Precise Binary Neural Network with Generalized Activation Functionscited by 183
  21. 2023HNeRV: A Hybrid Neural Representation for Videoscited by 100
  22. 2022LUT-GEMM: Quantized Matrix Multiplication based on LUTs for Efficient Inference in Large-Scale Generative Language Modelscited by 98
  23. 2023Memory-Efficient Fine-Tuning of Compressed Large Language Models via sub-4-bit Integer Quantizationcited by 97
  24. 2023LQ-LoRA: Low-rank Plus Quantized Matrix Decomposition for Efficient Language Model Finetuningcited by 92
  25. 2019Learning Fast Algorithms for Linear Transforms Using Butterfly Factorizationscited by 73
  26. 2023HiNeRV: Video Compression with Hierarchical Encoding-based Neural Representationcited by 70
  27. 2023QLLM: Accurate and Efficient Low-Bitwidth Quantization for Large Language Modelscited by 63
  28. 2013Estimating or Propagating Gradients Through Stochastic Neuronscited by 54
  29. 2024Diffusion-Based Neural Network Weights Generationcited by 38
  30. 2024ApiQ: Finetuning of 2-Bit Quantized Large Language Modelcited by 27
  31. 2024Mitigating the Impact of Outlier Channels for Language Model Quantization with Activation Regularizationcited by 23
  32. 2024Accurate Block Quantization in LLMs with Outlierscited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 1; 2026: 2.

3 papers.

  1. 2026CARE: Covariance-Aware and Rank-Enhanced Decomposition for Enabling Multi-Head Latent Attentioncited by 5
  2. 2025Neural Weight Compression for Language Modelscited by 0
  3. 2026TokenSeek: Memory Efficient Fine Tuning via Instance-Aware Token Ditchingcited by 0