mlsanityData to 13 Oct 2026

Squat: Quant Small Language Models on the Edge

arXiv 2402.10787 ↗First posted 16 Feb 2024cs.LG cs.AI cs.CLcited by 16

Not on the map. Why?

Abstract (arXiv)

A growing trend has emerged in designing high-quality Small Language Models (SLMs) with a few million parameters. This trend is driven by the increasing concerns over cloud costs, privacy, and latency. Considering that full parameter training is feasible for SLMs on mobile devices, Quantization-Aware Training (QAT) is employed to improve efficiency by reducing computational overhead and memory footprint. However, previous QAT works adopt fine-grained quantization methods to compress models with billions of parameters on GPUs, incompatible with current commodity hardware, such as mobile and edge devices, which relies on Single Instruction Multiple Data (SIMD) instructions. Thus, the generalization of these methods to SLMs on mobile devices is limited. In this paper, we propose Squat method, an effective QAT framework with deployable quantization for SLMs on mobile devices. Specifically, we propose entropy-guided and distribution-aligned distillation to mitigate the distortion of attention information from quantization. Besides, we employ sub-8-bit token adaptive quantization, assigning varying bit widths to different tokens based on their importance. Furthermore, we develop a SIMD-based Multi-Kernel Mixed-Precision (MKMP) multiplier to support sub-8-bit mixed-precision MAC on mobile devices. Our extensive experiments verify the substantial improvements of our method compared to other QAT methods across various datasets. Furthermore, we achieve an on-device speedup of up to 2.37x compared with its FP16 counterparts, signaling a great advancement. Code: https://github.com/shawnricecake/squant

Key facts. Cited by 16 papers in the map (data through 13 Oct 2026). Builds on 28 of its 55 references that are in the map. First posted on arXiv 16 Feb 2024.

Builds on

28 of 55 references.

References · 28

  1. 2020Language Models are Few-Shot Learnerscited by 32,320
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,072
  3. 2015Distilling the Knowledge in a Neural Networkcited by 10,180
  4. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,500
  5. 2013Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computationcited by 2,056
  6. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,884
  7. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,725
  8. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,365
  9. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,272
  10. 2019BLiMP: The Benchmark of Linguistic Minimal Pairs for Englishcited by 557
  11. 2022ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformerscited by 522
  12. 2018PACT: Parameterized Clipping Activation for Quantized Neural Networkscited by 465
16 more
  1. 2023LLM-QAT: Data-Free Quantization Aware Training for Large Language Modelscited by 322
  2. 2023SqueezeLLM: Dense-and-Sparse Quantizationcited by 292
  3. 2020SpAtten: Efficient Sparse Attention Architecture with Cascade Token and Head Pruningcited by 233
  4. 2021SPViT: Enabling Faster Vision Transformers via Soft Token Pruningcited by 166
  5. 2023Token Fusion: Bridging the Gap between Token Pruning and Token Mergingcited by 124
  6. 2022Compression of Generative Pre-trained Language Models via Quantizationcited by 79
  7. 2022BiBERT: Accurate Fully Binarized BERTcited by 72
  8. 2023Baby Llama: knowledge distillation from an ensemble of teachers trained on a small dataset with no performance penaltycited by 57
  9. 2022HeatViT: Hardware-Efficient Adaptive Token Pruning for Vision Transformerscited by 46
  10. 2023ZeroQuant-FP: A Leap Forward in LLMs Post-Training W4A8 Quantization Using Floating-Point Formatscited by 44
  11. 2021Achieving on-Mobile Real-Time Super-Resolution with Neural Architecture and Pruning Searchcited by 34
  12. 2023Agile-Quant: Activation-Guided Quantization for Faster Inference of LLMs on the Edgecited by 33
  13. 2022Pruning-as-Search: Efficient Neural Architecture Search via Channel Pruning and Structural Reparameterizationcited by 32
  14. 2022Compiler-Aware Neural Architecture Search for On-Mobile Real-time Super-Resolutioncited by 20
  15. 2023Token-Scaled Logit Distillation for Ternary Weight Generative Language Modelscited by 20
  16. 2022Accelerating Deep Learning Model Inference on Arm CPUs with Ultra-Low Bit Quantization and Runtimecited by 6

Built on by

Cited by year

Citing papers per year: 2024: 9; 2025: 7.

16 papers.

Sort:

By area:

  1. 2024Low-Rank Quantization-Aware Training for LLMscited by 43
  2. 2024A Survey of Small Language Modelscited by 31
  3. 2024Pruning Foundation Models for High Accuracy without Retrainingcited by 20
  4. 2024Rethinking Token Reduction for State Space Modelscited by 15
  5. 2024Search for Efficient Large Language Modelscited by 14
  6. 2024Fast and Memory-Efficient Video Diffusion Using Streamlined Inferencecited by 14
  7. 2024Designing Efficient LLM Accelerators for Edge Devicescited by 13
  8. 2025Structured Agent Distillation for Large Language Modelcited by 12
  9. 2025Distributed LLMs and Multimodal Large Language Models: A Survey on Advances, Challenges, and Future Directionscited by 10
  10. 2025QuartDepth: Post-Training Quantization for Real-Time Depth Estimation on the Edgecited by 8
  11. 2024EdgeShard: Efficient LLM Inference via Collaborative Edge Computingcited by 7
  12. 2025Universality of Layer-Level Entropy-Weighted Quantization Beyond Model Architecture and Sizecited by 4
4 more
  1. 2024Art and Science of Quantizing Large-Scale Models: A Comprehensive Overviewcited by 2
  2. 2025Quantitative Analysis of Performance Drop in DeepSeek Model Quantizationcited by 2
  3. 2025ROSAQ: Rotation-based Saliency-Aware Weight Quantization for Efficiently Compressing Large Language Modelscited by 2
  4. 2025Enabling Flexible Multi-LLM Integration for Scalable Knowledge Aggregationcited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2025SLMQuant:Benchmarking Small Language Model Quantization for Practical Deployment
  2. 2024MobileQuant: Mobile-friendly Quantization for On-device Language Models
  3. 2024Optimizing Large Language Models through Quantization: A Comparative Analysis of PTQ and QAT Techniques
  4. 2026Quant.npu: Enabling Efficient Mobile NPU Inference for on-device LLMs via Fully Static Quantization
  5. 2025End-to-End On-Device Quantization-Aware Training for LLMs at Inference Cost
  6. 2024Continuous Approximations for Improving Quantization Aware Training of LLMs
  7. 2024EfficientQAT: Efficient Quantization-Aware Training for Large Language Models
  8. 2026LC-QAT: Data-Efficient 2-Bit QAT for LLMs via Linear-Constrained Vector Quantization