mlsanityData to 7 Oct 2026

LoopQ: Quantization for Recursive Transformers

arXiv 2605.16343 ↗First posted 8 May 2026cs.LG cs.AIcited by 4

Not on the map. Why?

Abstract (arXiv)

Looped language models (LoopLMs) improve parameter efficiency by recursively reusing Transformer blocks, enabling deeper computation under a fixed model size. However, this reuse makes LoopLMs more fragile under post-training quantization (PTQ). We present the first systematic study of quantization in LoopLMs and identify three challenges: distribution shift across roles, state reuse across loop transitions, and recursive error accumulation. To address these challenges, we propose LoopQ, a loop-aware PTQ framework that preserves a shared quantized backbone while introducing lightweight adaptations. LoopQ combines activation scaling, selective transformation, cross-loop state alignment, and trajectory-aware optimization to reduce distributional mismatch within loops and error accumulation across loops. Experiments across seven benchmarks show that, under W4A4 quantization, LoopQ improves average downstream accuracy by 68.8% and reduces average perplexity by 87.7% compared with the strongest static PTQ baseline.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Builds on 37 of its 44 references that are in the map. First posted on arXiv 8 May 2026.

Builds on

37 of 44 references.

References · 37

  1. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  2. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  3. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  4. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  5. 2016Pointer Sentinel Mixture Modelscited by 2,992
  6. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
  7. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  8. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,838
  9. 2019Root Mean Square Layer Normalizationcited by 1,425
  10. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,338
  11. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,234
  12. 2016The LAMBADA dataset: Word prediction requiring a broad discourse contextcited by 960
25 more
  1. 2015Optimizing Neural Networks with Kronecker-factored Approximate Curvaturecited by 776
  2. 2024QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMscited by 598
  3. 2024SpinQuant: LLM quantization with learned rotationscited by 446
  4. 2025Scaling Latent Reasoning via Looped Language Modelscited by 145
  5. 2025Mixture-of-Recursions: Learning Dynamic Recursive Depths for Adaptive Token-Level Computationcited by 120
  6. 2024FlatQuant: Flatness Matters for LLM Quantizationcited by 116
  7. 2025OstQuant: Refining Large Language Model Quantization with Orthogonal and Scaling Transformations for Better Distribution Fittingcited by 94
  8. 2024Looped Transformers for Length Generalizationcited by 91
  9. 2020Robust Quantization: One Model to Rule Them Allcited by 63
  10. 2024SliM-LLM: Salience-Driven Mixed-Precision Quantization for Large Language Modelscited by 61
  11. 2024Exploiting LLM Quantizationcited by 61
  12. 2026LoopFormer: Elastic-Depth Looped Transformers for Latent Reasoning via Shortcut Modulationcited by 48
  13. 2026Parcae: Scaling Laws For Stable Looped Language Modelscited by 45
  14. 2023Transformer Fusion with Optimal Transportcited by 35
  15. 2024ResQ: Mixed-Precision Quantization of Large Language Models with Low-Rank Residualscited by 32
  16. 2023RobustMQ: Benchmarking Robustness of Quantized Modelscited by 19
  17. 2024Pruning via Merging: Compressing LLMs via Manifold Alignment Based Layer Mergingcited by 18
  18. 2026Hyperloop Transformerscited by 11
  19. 2026Pushing the Limits of Block Rotations in Post-Training Quantizationcited by 9
  20. 2023Modular Transformers: Compressing Transformers into Modularized Layers for Flexible Efficient Inferencecited by 4
  21. 2024PIMphony: Overcoming Bandwidth and Capacity Inefficiency in PIM-based Long-Context LLM Inference Systemcited by 4
  22. 2026TurboBoA: Faster and Exact Attention-aware Quantization without Backpropagationcited by 2
  23. 2026A Persistent-State Dataflow Accelerator for Memory-Bound Linear Attention Decode on FPGAcited by 2
  24. 2021CREW: Computation Reuse and Efficient Weight Storage for Hardware-accelerated MLPs and RNNscited by 1
  25. 2025Merging Feed-Forward Sublayers for Compressed Transformerscited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 4.

4 papers.

  1. 2026Q-DEQ: Discrete Solving and Quantization for Deep Equilibrium Models in Time Series Forecasting under Edge Deployment Coding Constraintscited by 0
  2. 2026Quantizing Looped Transformers: Feedback Exposure and Calibration Blindnesscited by 0
  3. 2026A Tilted Bowl Is Not a Slippery Slope: Compressing Looped Modelscited by 0
  4. 2026ResidualQuant: KV Cache Quantization for Looped Transformers with 2-Bit Residualscited by 0