mlsanityData to 7 Oct 2026

LoRaQ: Optimized Low Rank Approximation for 4-bit Quantization

arXiv 2604.18117 ↗First posted 20 Apr 2026cs.LGcited by 2

On the map: Diffusion models → Diffusion foundations and sampling

Abstract (arXiv)

Post-training quantization (PTQ) is essential for deploying large diffusion transformers on resource-constrained hardware, but aggressive 4-bit quantization significantly degrades generative performance. Low-rank approximation methods have emerged as a promising solution by appending auxiliary linear branches to restore performance. However, current state-of-the-art approaches assume these branches must retain high precision (W16A16) and rely on heavy, data-dependent calibration for initialization. We challenge both limitations with LoRaQ (Low-Rank Approximated Quantization), a simple, data-free calibration approach that optimizes quantization error compensation. By overcoming the need for high-precision branches, LoRaQ enables the first fully sub-16 bit pipeline, allowing the low-rank branch itself to be quantized. We demonstrate that, at equal memory overhead, LoRaQ outperforms the state-of-the-art methods in their native implementations on Pixart-Sigma and SANA. We also analyze mixed-precision configurations, showing that setups such as W8A8, W6A6, and W4A8 for the low-rank branch, alongside a W4 main layer, yield superior results while maintaining a fully quantized architecture compatible with modern mixed-precision hardware.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in Diffusion foundations and sampling, one of 14 directions in Diffusion models. Builds on 24 of its 37 references that are in the map. First posted on arXiv 20 Apr 2026.

Builds on

24 of 37 references.

Closest ancestors · 12

Same direction, newest first

  1. 2024TCAQ-DM: Timestep-Channel Adaptive Quantization for Diffusion Models21 Dec
  2. 2024Q-DiT: Accurate Post-Training Quantization for Diffusion Transformers25 Jun
  3. 2024MixDQ: Memory-Efficient Few-Step Text-to-Image Diffusion Models with Metric-Decoupled Mixed Precision Quantization28 May
  4. 2024PTQ4DiT: Post-training Quantization for Diffusion Transformers25 May
  5. 2023Efficient Quantization Strategies for Latent Diffusion Models9 Dec
  6. 2023Post-training Quantization for Text-to-Image Diffusion Models with Progressive Calibration and Activation Relaxing10 Nov
  7. 2023PTQD: Accurate Post-Training Quantization for Diffusion Models18 May
  8. 2023Q-Diffusion: Quantizing Diffusion Models8 Feb
  9. 2022Scalable Diffusion Models with Transformers19 Dec
  10. 2022Post-training Quantization on Diffusion Models28 Nov
  11. 2022All are Worth Words: A ViT Backbone for Diffusion Models25 Sep
  12. 2020Denoising Diffusion Probabilistic Models19 Jun

Other references · 12

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,029
  3. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  4. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,338
  5. 2018Demystifying MMD GANscited by 1,268
  6. 2023ImageReward: Learning and Evaluating Human Preferences for Text-to-Image Generationcited by 1,010
  7. 2024QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMscited by 598
  8. 2024SpinQuant: LLM quantization with learned rotationscited by 446
  9. 2024QuIP#: Even Better LLM Quantization with Hadamard Incoherence and Lattice Codebookscited by 354
  10. 2022Outlier Suppression: Pushing the Limit of Low-bit Transformer Language Modelscited by 176
  11. 2024SageAttention2: Efficient Attention with Thorough Outlier Smoothing and Per-thread INT4 Quantizationcited by 113
  12. 2023A Picture is Worth More Than 77 Text Tokens: Evaluating CLIP-Style Models on Dense Captionscited by 107

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026KroQuant: Kronecker-Structured Block Transforms for Efficient Post-Training Quantization of Diffusion Transformerscited by 0
  2. 2026Deflating the Hessian: Rank-4 W4A4 Quantization for Multimodal Diffusion Transformerscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026LoPRo: Enhancing Low-Rank Quantization via Permuted Block-Wise Rotation
  2. 2025TreeQ: Pushing the Quantization Boundary of Diffusion Transformer via Tree-Structured Mixed-Precision Search
  3. 2025LRQ-DiT: Log-Rotation Post-Training Quantization of Diffusion Transformers for Image and Video Generation
  4. 2023EfficientDM: Efficient Quantization-Aware Fine-Tuning of Low-Bit Diffusion Models
  5. 2025LoRAQuant: Mixed-Precision Quantization of LoRA to Ultra-Low Bits
  6. 2026Breaking the Blocks: Continuous Low-Rank Decomposed Scaling for Unified LLM Quantization and Adaptation
  7. 2024QuEST: Low-bit Diffusion Model Quantization via Efficient Selective Finetuning
  8. 2026Recover-LoRA for Aggressive Quantization: Reclaiming Accuracy in 2-Bit Language Models via Low-Rank Adaptation with Knowledge Distillation on Synthetic Data