dMX: Differentiable Mixed-Precision Assignment for Low-Precision Floating-Point Formats
Not on the map. Why?
Abstract (arXiv)
Quantizing large language models (LLMs) to low-precision floating-point representations is central to efficient deployment, yet applying a single bit-width uniformly across all layers is sub-optimal in terms of both performance and accuracy. This work introduces dMX, a differentiable mixed-precision quantization framework for learnable floating-point bit-width assignment. We study its application for the microscaling floating-point (MXFP) family of data types defined by the Open Compute Project (OCP) standard. The per-layer bit-width assignment is formulated as a continuous optimization problem in which each layer's floating-point format format is parameterized by a scalar parameter, folding the multi-variate design space into a single learnable offset. During training this offset takes continuous values, avoiding sudden oscillations between discrete quantization formats. A temperature-based annealing schedule progressively discretizes the learned offsets, ensuring that the final configuration maps to hardware-compatible MXFP formats without abrupt transitions between training and inference behavior. A target-aware regularization term steers the average bit-width toward a user-specified budget, serving as a coarse-grained proxy for inference cost and balancing model quality against deployment efficiency. We performed experiments on different families of LLM, such as Llama, Qwen3, and SmolLM2, evaluating perplexity on WikiText-2 and accuracy on four zero-shot reasoning benchmarks. Across these settings, dMX consistently yields Pareto-dominating models and improves over Kullback-Leibler (KL) divergence-based layer-selection heuristics, efficiently navigating trade-offs between model quality and average bit-width.
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 39 of its 45 references that are in the map. First posted on arXiv 2 Jun 2026.
Builds on
39 of 45 references.
References · 39
- 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
- 2024The Llama 3 Herd of Modelscited by 14,192
- 2025Qwen3 Technical Reportcited by 8,195
- 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
- 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
- 2016Pointer Sentinel Mixture Modelscited by 2,992
- 2016Categorical Reparameterization with Gumbel-Softmaxcited by 2,708
- 2013Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computationcited by 2,031
- 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,838
- 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,338
- 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,234
- 2016The Concrete Distribution: A Continuous Relaxation of Discrete Random Variablescited by 1,135
27 more
- 2024The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scalecited by 1,104
- 2024SpinQuant: LLM quantization with learned rotationscited by 446
- 2018HAQ: Hardware-Aware Automated Quantization with Mixed Precisioncited by 426
- 2019HAWQ: Hessian AWare Quantization of Neural Networks with Mixed-Precisioncited by 387
- 2025SmolLM2: When Smol Goes Big -- Data-Centric Training of a Small Language Modelcited by 307
- 2019HAWQ-V2: Hessian Aware trace-Weighted Quantization of Neural Networkscited by 250
- 2022FP8 Formats for Deep Learningcited by 187
- 2023Microscaling Data Formats for Deep Learningcited by 132
- 2018Mixed Precision Quantization of ConvNets via Differentiable Neural Architecture Searchcited by 122
- 2020Bayesian Bits: Unifying Quantization and Pruningcited by 93
- 2019Trained Quantization Thresholds for Accurate and Efficient Fixed-Point Inference of Deep Neural Networkscited by 87
- 2020Rethinking Differentiable Search for Mixed-Precision Neural Networkscited by 79
- 2021Towards Mixed-Precision Quantization of Neural Networks via Constrained Optimizationcited by 57
- 2020FracBits: Mixed Precision Quantization via Fractional Bit-Widthscited by 53
- 2020HMQ: Hardware Friendly Mixed Precision Quantization Block for CNNscited by 43
- 2024Mixture Compressor for Mixture-of-Experts LLMs Gains Morecited by 38
- 2021BSQ: Exploring Bit-Level Sparsity for Mixed-Precision Neural Network Quantizationcited by 37
- 2022Q-ViT: Fully Differentiable Quantization for Vision Transformercited by 36
- 2024APTQ: Attention-aware Post-Training Mixed-Precision Quantization for Large Language Modelscited by 33
- 2024ResQ: Mixed-Precision Quantization of Large Language Models with Low-Rank Residualscited by 32
- 2023FP8 versus INT8 for efficient deep learning inferencecited by 31
- 2022SDQ: Stochastic Differentiable Quantization with Mixed Precisioncited by 27
- 2025MicroMix: Efficient Mixed-Precision Quantization with Microscaling Formats for Large Language Modelscited by 11
- 2026Pushing the Limits of Block Rotations in Post-Training Quantizationcited by 9
- 2025Mix-QSAM: Mixed-Precision Quantization of the Segment Anything Modelcited by 4
- 2025InfoQ: Mixed-Precision Quantization via Global Information Flowcited by 4
- 2024Gradient-Free Training of Quantized Neural Networkscited by 1
Built on by
Citing papers per year: 2026: 2.
2 papers.