mlsanityData to 7 Oct 2026

SoloQ: Calibration-Free Quantization for Diffusion Language Models

arXiv 2610.07121 ↗First posted 5 Oct 2026cs.LGcited by 0

On the map: Diffusion models → Discrete and text diffusion

also in 1 more
Abstract (arXiv)

Diffusion large language models dLLMs) have emerged as a promising alternative to autoregressive language models through bidirectional diffusion-based token generation. However, their growing model sizes and high inference costs make efficient deployment challenging: full-sequence denoising repeatedly invokes compute-intensive forward passes, while block-diffusion models additionally introduce a memory-intensive KV-cache. Low-bit weight-activation quantization is therefore attractive, yet existing dLLM post-training quantization methods rely on calibration data despite activation distributions shifting across masking states and denoising steps. We present SoloQ, a calibration-free quantization framework that maps weights and activations into a normalized rotated basis with a predictable marginal distribution, enabling data-independent quantization. SoloQ combines a structured K-RPBH rotation with a lightweight rescaling correction for calibration-free quantization. Its predictable post-rotation distribution supports both distribution-matched codebooks and hardware-native NVFP4. For block-diffusion models, SoloQ further applies commit-time KV-cache quantization to compress persistent states without perturbing the actively denoised block. Across full-sequence dLLMs (LLaDA and Dream) and block-diffusion dLLMs(Fast-dLLM v2 and Nemotron-Labs-Diffusion), SoloQ retains accuracy under 4-bit quantization and outperforms calibration-based baselines on knowledge- and reasoning-intensive benchmarks. With NVFP4, SoloQ reduces peak memory by up to 2.61X and accelerates end-to-end inference by up to 2.24X.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Discrete and text diffusion, one of 14 directions in Diffusion models. Builds on 61 of its 64 references that are in the map. First posted on arXiv 5 Oct 2026.

Builds on

61 of 64 references.

Closest ancestors · 6

Same direction, newest first

  1. 2026Nemotron-Labs-TwoTower: Diffusion Language Modeling with Pretrained Autoregressive Context25 Jun
  2. 2025Attention Sinks in Diffusion Language Models17 Oct
  3. 2025Dream 7B: Diffusion Large Language Models21 Aug
  4. 2025Block Diffusion: Interpolating Between Autoregressive and Diffusion Language Models12 Mar
  5. 2025Large Language Diffusion Models14 Feb
  6. 2022DiffusionBERT: Improving Generative Masked Language Models with Diffusion Models28 Nov

Other references · 55

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  5. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  6. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  7. 2022Scalable Diffusion Models with Transformerscited by 5,931
  8. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  9. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  10. 2024Scaling Rectified Flow Transformers for High-Resolution Image Synthesiscited by 4,094
  11. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  12. 2021Program Synthesis with Large Language Modelscited by 3,509
38 more
  1. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
  2. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
  3. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,748
  4. 2024CogVideoX: Text-to-Video Diffusion Models with An Expert Transformercited by 2,198
  5. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  6. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,838
  7. 2021Structured Denoising Diffusion Models in Discrete State-Spacescited by 1,665
  8. 2024HunyuanVideo: A Systematic Framework For Large Video Generative Modelscited by 1,597
  9. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,338
  10. 2023Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generationcited by 1,332
  11. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,234
  12. 2023Instruction-Following Evaluation for Large Language Modelscited by 1,143
  13. 2022Diffusion-LM Improves Controllable Text Generationcited by 1,006
  14. 2025Qwen-Image Technical Reportcited by 973
  15. 2023C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Modelscited by 694
  16. 2024KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cachecited by 604
  17. 2024QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMscited by 598
  18. 2022All are Worth Words: A ViT Backbone for Diffusion Modelscited by 498
  19. 2024SpinQuant: LLM quantization with learned rotationscited by 446
  20. 2025Fast-dLLM: Training-free Acceleration of Diffusion LLM by Enabling KV Cache and Parallel Decodingcited by 404
  21. 2023OmniQuant: Omnidirectionally Calibrated Quantization for Large Language Modelscited by 403
  22. 2023QuIP: 2-Bit Quantization of Large Language Models With Guaranteescited by 397
  23. 2024QuIP#: Even Better LLM Quantization with Hadamard Incoherence and Lattice Codebookscited by 354
  24. 2025LLaDA 1.5: Variance-Reduced Preference Optimization for Large Language Diffusion Modelscited by 273
  25. 2025Fast-dLLM v2: Efficient Block-Diffusion LLMcited by 128
  26. 2025TurboQuant: Online Vector Quantization with Near-optimal Distortion Ratecited by 116
  27. 2025OstQuant: Refining Large Language Model Quantization with Orthogonal and Scaling Transformations for Better Distribution Fittingcited by 94
  28. 2023Outlier Suppression+: Accurate quantization of large language models by equivalent and optimal shifting and scalingcited by 62
  29. 2025GPTAQ: Efficient Finetuning-Free Quantization for Asymmetric Calibrationcited by 54
  30. 2021EDEN: Communication-Efficient and Robust Distributed Mean Estimation for Federated Learningcited by 53
  31. 2021DRIVE: One-bit Distributed Mean Estimationcited by 52
  32. 2024RaBitQ: Quantizing High-Dimensional Vectors with a Theoretical Error Bound for Approximate Nearest Neighbor Searchcited by 46
  33. 2025PolarQuant: Quantizing KV Caches with Polar Transformationcited by 24
  34. 2026Nemotron-Labs-Diffusion: A Tri-Mode Language Model Unifying Autoregressive, Diffusion, and Self-Speculation Decodingcited by 19
  35. 2025DVD-Quant: Data-free Video Diffusion Transformers Quantizationcited by 14
  36. 2025Quant-dLLM: Post-Training Extreme Low-Bit Quantization for Diffusion Large Language Modelscited by 11
  37. 2025DLLMQuant: Quantizing Diffusion-based Large Language Modelscited by 10
  38. 2025ConvRot: Rotation-Based Plug-and-Play 4-bit Quantization for Diffusion Transformerscited by 5

50 most cited of 55 All 55 (JSON)

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026On the Quantization Robustness of Diffusion Language Models in Coding Benchmarks
  2. 2025FlashDLM: Accelerating Diffusion Language Model Inference via Efficient KV Caching and Guided Diffusion
  3. 2026LESS Is More: Mutual-Stability Sampling for Diffusion Language Models
  4. 2026CAI-DLLM: Convergence Aware Inference for Diffusion Language Models
  5. 2026Window-Diffusion: Accelerating Diffusion Language Model Inference with Windowed Token Pruning and Caching
  6. 2026Uncertainty Quantification for Large Language Diffusion Models
  7. 2025dInfer: An Efficient Inference Framework for Diffusion Language Models
  8. 2025Pioneering 4-Bit FP Quantization for Diffusion Models: Mixup-Sign Quantization and Timestep-Aware Fine-Tuning