mlsanityData to 7 Oct 2026

Scale-QLoRA: Code-Invariant Adapter Merging for Native 4-bit Microscaling LLMs

arXiv 2609.04526 ↗First posted 3 Sep 2026cs.CL cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Merging a LoRA adapter into its base model is standard deployment practice: it removes the runtime adapter's per-forward overhead and leaves a single standalone checkpoint any serving stack can load. On a native 4-bit microscaling checkpoint (NVFP4, MXFP4) that step stops being free. The merged weights must be written back through a quantizer, which re-derives the checkpoint's discrete E2M1 code plane (roughly 90% of the artifact's bytes), so the deployed artifact becomes coupled to one quantization convention, and every later code-touching event in its lifecycle can move it. Done naively the step is worse than fragile: it deletes the adaptation, by up to 39 pp, because against an already-on-grid base the reconstruction optimum is that base. Scale-QLoRA instead adapts only the native per-block scale field, trains those scales on the deployment grid, and freezes every E2M1 code. Within a fixed native format, scale grid, block layout and code plane, merging is then a bit-exact identity and the merged artifact is code-invariant. Across four models and four tasks, Scale-QLoRA and merge-aware QAT-LoRA are both accuracy-lossless, so we claim no accuracy ordering between them; they differ structurally, in that QAT-LoRA re-derives the code plane through a quantizer while Scale-QLoRA preserves it exactly. That difference is what the lifecycle prices: nearest-rounding implementations disagree by about a point on the measured task, and more extreme rule mismatches can drive the weight-space artifact to ~0%, which we report as a sensitivity bound rather than a deployment frequency. Preserving the code plane also drops the weight-space straight-through estimator from training (3.9x per step on the dense 8B model) and enables exact rollback, code-plane deduplication, and a ~125x faster scale-only task swap.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 26 of its 35 references that are in the map. First posted on arXiv 3 Sep 2026.

Builds on

26 of 35 references.

References · 26

  1. 2024The Llama 3 Herd of Modelscited by 14,192
  2. 2025Qwen3 Technical Reportcited by 8,195
  3. 2021Program Synthesis with Large Language Modelscited by 3,509
  4. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,838
  5. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,234
  6. 2018Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Taskcited by 1,120
  7. 2018PACT: Parameterized Clipping Activation for Quantized Neural Networkscited by 456
  8. 2020Efficient Intent Detection with Dual Sentence Encoderscited by 419
  9. 2022DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scalecited by 400
  10. 2019An Evaluation Dataset for Intent Classification and Out-of-Scope Predictioncited by 378
  11. 2023LoftQ: LoRA-Fine-Tuning-Aware Quantization for Large Language Modelscited by 195
  12. 2020LSQ+: Improving low-bit quantization through learnable offsets and better initializationcited by 180
14 more
  1. 2023S-LoRA: Serving Thousands of Concurrent LoRA Adapterscited by 146
  2. 2019Additive Powers-of-Two Quantization: An Efficient Non-uniform Discretization for Neural Networkscited by 143
  3. 2023Microscaling Data Formats for Deep Learningcited by 132
  4. 2024EfficientQAT: Efficient Quantization-Aware Training for Large Language Modelscited by 123
  5. 2023Memory-Efficient Fine-Tuning of Compressed Large Language Models via sub-4-bit Integer Quantizationcited by 97
  6. 2023LQ-LoRA: Low-rank Plus Quantized Matrix Decomposition for Efficient Language Model Finetuningcited by 92
  7. 2023Punica: Multi-Tenant LoRA Servingcited by 80
  8. 2025Optimizing Large Language Model Training Using FP4 Quantizationcited by 53
  9. 2024Low-Rank Quantization-Aware Training for LLMscited by 42
  10. 2024Accurate LoRA-Finetuning Quantization of LLMs via Information Retentioncited by 40
  11. 2024L4Q: Parameter Efficient Quantization-Aware Fine-Tuning on Large Language Modelscited by 15
  12. 2026ARCQuant: Boosting NVFP4 Quantization with Augmented Residual Channels for LLMscited by 14
  13. 2025LoTA-QAF: Lossless Ternary Adaptation for Quantization-Aware Fine-Tuningcited by 3
  14. 2026Breaking the Blocks: Continuous Low-Rank Decomposed Scaling for Unified LLM Quantization and Adaptationcited by 1

Built on by

Not cited yet.