mlsanityData to 14 Oct 2026

UltraSketchLLM: Sub-1-Bit LLM Compression via Sketch and Hardware-Friendly Operators

arXiv 2506.17255 ↗First posted 8 Jun 2025cs.LG cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

Large language models (LLMs) require larger GPU memory size these days, necessitating efficient and extreme weight compression methods. Existing compression methods are either theoretically limited by 1 bit per weight or face severe performance degradation and inefficiency. To deploy LLMs in resource-constrained scenarios, we introduce UltraSketchLLM, compressing LLMs with data sketch. It reduces peak GPU memory footprint with a high compression rate down to 0.5 bit per weight. Combined with hardware-friendly implementation, UltraSketchLLM keeps tolerable performance degradation and extremely low latency overhead with 14.9x speedup compared to naive sketch solution.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 17 of its 29 references that are in the map. First posted on arXiv 8 Jun 2025.

Builds on

17 of 29 references.

References · 17

  1. 2017Attention Is All You Needcited by 60,361
  2. 2024The Llama 3 Herd of Modelscited by 14,715
  3. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  4. 2016Pointer Sentinel Mixture Modelscited by 3,096
  5. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,883
  6. 2015Character-level Convolutional Networks for Text Classificationcited by 2,330
  7. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,218
  8. 2013Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computationcited by 2,079
  9. 2020Transformer Feed-Forward Layers Are Key-Value Memoriescited by 1,336
  10. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,298
  11. 2022LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scalecited by 906
  12. 2023Atom: Low-bit Quantization for Efficient and Accurate LLM Servingcited by 272
5 more
  1. 2024Extreme Compression of Large Language Models via Additive Quantizationcited by 232
  2. 2015Compressing Neural Networks with the Hashing Trickcited by 150
  3. 2024What Makes Quantization for Large Language Models Hard? An Empirical Study from the Lens of Perturbationcited by 23
  4. 2025HACK: Homomorphic Acceleration via Compression of the Key-Value Cache for Disaggregated LLM Inferencecited by 10
  5. 2022Efficient model compression with Random Operation Access Specific Tile (ROAST) hashingcited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Huff-LLM: End-to-End Lossless Compression for Efficient LLM Inference
  2. 2026ZipServ: Fast and Memory-Efficient LLM Inference with Hardware-Aware Lossless Compression
  3. 2023Efficient LLM inference solution on Intel GPU
  4. 2023Compressing LLMs: The Truth is Rarely Pure and Never Simple
  5. 2025ClusComp: A Simple Paradigm for Model Compression and Efficient Finetuning
  6. 2026SplitZip: Ultra Fast Lossless KV Compression for Disaggregated LLM Serving
  7. 2024FlightLLM: Efficient Large Language Model Inference with a Complete Mapping Flow on FPGAs
  8. 2023FlashDecoding++: Faster Large Language Model Inference on GPUs