mlsanityData to 7 Oct 2026

Rethinking RoPE Scaling in Quantized LLM: Theory, Outlier, and Channel-Band Analysis with Weight Rescaling

arXiv 2510.00028 ↗First posted 26 Sep 2025cs.LG cs.AIcited by 2

On the map: LLM reasoning → Test-time compute and efficient thinking

Abstract (arXiv)

Extending the context window support of large language models (LLMs) is crucial for tasks with long-distance dependencies. RoPE-based interpolation and extrapolation methods, such as linear scaling and frequency-aware schemes, enable longer input length support without retraining, while post-training quantization (PTQ) makes deployment practical. However, we show that combining RoPE position interpolation (PI) with PTQ degrades accuracy due to coupled effects including long-context aliasing, dynamic-range dilation, anisotropy from axis-aligned quantizers vs. rotated RoPE pairs, and outlier shifting that produces position-dependent logit noise. We provide, to the best of our knowledge, the first systematic analysis of the PI+PTQ approach and introduce two practical diagnostics: interpolation pressure (per-band sensitivity to phase scaling) and tail-inflation ratios (outlier shift from short to long contexts). Following the analysis results, we propose Q-ROAR (Quantization, RoPE-interpolation, and Outlier Aware Rescaling), a weight-only, interpolation-aware stabilization of PI for quantized LLMs. Q-ROAR groups RoPE dimensions into a small number of frequency bands and performs a lightweight search over per-band scales for Key and Query weights (with an optional symmetric variant to preserve logit scale). The search is guided by our diagnostics and uses a tiny long-context development dataset, requiring no fine-tuning to the model, no architecture or kernel changes, and no additional deployment overhead. Empirically, Q-ROAR reduces the model's perplexity on long-context workloads by more than 14%, while preserving short-context performance, inference throughput, and compatibility with existing LLM system stacks.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in Test-time compute and efficient thinking, one of 17 directions in LLM reasoning. Builds on 27 of its 30 references that are in the map. First posted on arXiv 26 Sep 2025.

Builds on

27 of 30 references.

References · 27

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  5. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  6. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,146
  7. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  8. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  9. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  10. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,383
  11. 2016Pointer Sentinel Mixture Modelscited by 2,992
  12. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,140
15 more
  1. 2023Code Llama: Open Foundation Models for Codecited by 2,105
  2. 2024Qwen2 Technical Reportcited by 2,076
  3. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,838
  4. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,338
  5. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,234
  6. 2021Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationcited by 1,085
  7. 2023Extending Context Window of Large Language Models via Positional Interpolationcited by 721
  8. 2024QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMscited by 598
  9. 2023YaRN: Efficient Context Window Extension of Large Language Modelscited by 584
  10. 2024SpinQuant: LLM quantization with learned rotationscited by 446
  11. 2021Efficient Attentions for Long Document Summarizationcited by 342
  12. 2024LongRoPE: Extending LLM Context Window Beyond 2 Million Tokenscited by 302
  13. 2023LongLoRA: Efficient Fine-tuning of Long-Context Large Language Modelscited by 243
  14. 2024How to Train Long-Context Language Models (Effectively)cited by 141
  15. 2024LongAlign: A Recipe for Long Context Alignment of Large Language Modelscited by 102

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026Rotation-Aligned Key Channel Pruning for Efficient Vision-Language Model Inferencecited by 0
  2. 2026When Local Variance Optimality Is Not Enough: RoPE-Aligned Q/K Rotations for Dynamic 4-Bit Quantisationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Why Do Some Inputs Break Low-Bit LLM Quantization?
  2. 2026Rethinking Residual Errors in Compensation-based LLM Quantization
  3. 2023Rethinking Channel Dimensions to Isolate Outliers for Low-bit Weight Quantization of Large Language Models
  4. 2025FireQ: Fast INT4-FP8 Kernel and RoPE-aware Quantization for LLM Inference Acceleration
  5. 2026C-PTQ: Fisher-weighted Channel-wise Sensitivity for Post-training Quantization of MLLMs
  6. 2026SliderQuant: Accurate Post-Training Quantization for LLMs
  7. 2024Taming Sensitive Weights : Noise Perturbation Fine-tuning for Robust LLM Quantization
  8. 2025Rethinking the Outlier Distribution in Large Language Models: An In-depth Study