mlsanityData to 14 Oct 2026

One-Bit Quantization for Random Features Models

arXiv 2510.16250 ↗First posted 17 Oct 2025cs.LG stat.MLcited by 0

Not on the map. Why?

Abstract (arXiv)

Recent advances in neural networks have led to significant computational and memory demands, spurring interest in one-bit weight compression to enable efficient inference on resource-constrained devices. However, the theoretical underpinnings of such compression remain poorly understood. We address this gap by analyzing one-bit quantization in the Random Features model, a simplified framework that corresponds to neural networks with random representations. We prove that, asymptotically, quantizing weights of all layers except the last incurs no loss in generalization error, compared to the full precision random features model. Our findings offer theoretical insights into neural network compression. We also demonstrate empirically that one-bit quantization leads to significant inference speed ups for the Random Features models even on a laptop GPU, confirming the practical benefits of our work. Additionally, we provide an asymptotically precise characterization of the generalization error for Random Features with an arbitrary number of layers. To the best of our knowledge, our analysis yields more general results than all previous works in the related literature.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 31 of its 47 references that are in the map. First posted on arXiv 17 Oct 2025.

Builds on

31 of 47 references.

References · 31

  1. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,621
  2. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,919
  3. 2019Surprises in High-Dimensional Ridgeless Least Squares Interpolationcited by 513
  4. 2023QuIP: 2-Bit Quantization of Large Language Models With Guaranteescited by 414
  5. 2019The generalization error of random features regression: Precise asymptotics and double descent curvecited by 413
  6. 2024The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bitscited by 308
  7. 2023BitNet: Scaling 1-bit Transformers for Large Language Modelscited by 189
  8. 2019Linearized two-layers neural networks in high dimensioncited by 155
  9. 2020Generalisation error in learning with random features and the hidden manifold modelcited by 127
  10. 2020The Gaussian equivalence of generative models for learning with shallow neural networkscited by 113
  11. 2024A Survey of Resource-efficient LLM and Multimodal Foundation Modelscited by 86
  12. 2022Universality of empirical risk minimizationcited by 75
19 more
  1. 2023A Theory of Non-Linear Feature Learning with One Gradient Step in Two-Layer Neural Networkscited by 49
  2. 2020Random Matrix Theory Proves that Deep Learning Representations of GAN-data Behave as Gaussian Mixturescited by 47
  3. 2020A Precise High-Dimensional Asymptotic Theory for Boosting and Minimum-ℓ1-Norm Interpolated Classifierscited by 44
  4. 2022The curse of overparametrization in adversarial training: Precise analysis of robust generalization for random features regressioncited by 41
  5. 2023Bayes-optimal Learning of Deep Random Networks of Extensive-widthcited by 41
  6. 2019Stochastic Mirror Descent on Overparameterized Nonlinear Models: Convergence, Implicit Regularization, and Generalizationcited by 40
  7. 2020Provable Benefits of Overparameterization in Model Compression: From Double Descent to Pruning Neural Networkscited by 40
  8. 2023What can a Single Attention Layer Learn? A Study Through the Random Features Lenscited by 40
  9. 2023Deterministic equivalent and error universality of deep random features learningcited by 35
  10. 2019The generalization error of max-margin linear classifiers: Benign overfitting and high dimensional asymptotics in the overparametrized regimecited by 28
  11. 2023Universality laws for Gaussian mixtures in generalized linear modelscited by 28
  12. 2023Precise Asymptotic Analysis of Deep Random Feature Modelscited by 22
  13. 2015Universality laws for randomized dimension reduction, with applicationscited by 21
  14. 2023Learning curves for deep structured Gaussian feature modelscited by 13
  15. 2024Asymptotics of Learning with Deep Structured (Random) Featurescited by 13
  16. 2023Regularized Linear Regression for Binary Classificationcited by 7
  17. 2024QGen: On the Ability to Generalize in Quantization Aware Trainingcited by 7
  18. 2024Universality in Transfer Learning for Linear Modelscited by 6
  19. 2024A Novel Gaussian Min-Max Theorem and its Applicationscited by 4

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2022A simple approach for quantizing neural networks
  2. 2021A Survey of Quantization Methods for Efficient Neural Network Inference
  3. 2025Low-bit Model Quantization for Deep Neural Networks: A Survey
  4. 2024When are 1.58 bits enough? A Bottom-up Exploration of BitNet Quantization
  5. 2026Quantization-Aware Regularizers for Deep Neural Networks Compression
  6. 2022On One-Bit Quantization
  7. 2022Post-training Quantization for Neural Networks with Provable Guarantees
  8. 2024SLiM: One-shot Quantization and Sparsity with Low-rank Approximation for LLM Weight Compression