mlsanityData to 14 Oct 2026

On Expressive Power of Quantized Neural Networks under Fixed-Point Arithmetic

arXiv 2409.00297 ↗First posted 30 Aug 2024cs.LG stat.MLcited by 4

Not on the map. Why?

Abstract (arXiv)

Existing works on the expressive power of neural networks typically assume real parameters and exact operations. In this work, we study the expressive power of quantized networks under discrete fixed-point parameters and inexact fixed-point operations with round-off errors. We first provide a necessary condition and a sufficient condition on fixed-point arithmetic and activation functions for quantized networks to represent all fixed-point functions from fixed-point vectors to fixed-point numbers. Then, we show that various popular activation functions satisfy our sufficient condition, e.g., Sigmoid, ReLU, ELU, SoftPlus, SiLU, Mish, and GELU. In other words, networks using those activation functions are capable of representing all fixed-point functions. We further show that our necessary condition and sufficient condition coincide under a mild condition on activation functions: e.g., for an activation function σ, there exists a fixed-point number x such that σ(x)=0. Namely, we find a necessary and sufficient condition for a large class of activation functions. We lastly show that even quantized networks using binary weights in -1,1 can also represent all fixed-point functions for practical activation functions.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Builds on 24 of its 39 references that are in the map. First posted on arXiv 30 Aug 2024.

Builds on

24 of 39 references.

References · 24

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2018MobileNetV2: Inverted Residuals and Linear Bottleneckscited by 4,795
  3. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,307
  4. 2017Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inferencecited by 1,593
  5. 2016Error bounds for approximations with deep ReLU networkscited by 574
  6. 2024SpinQuant: LLM quantization with learned rotationscited by 471
  7. 2016Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activationscited by 426
  8. 2019Are Transformers universal approximators of sequence-to-sequence functions?cited by 406
  9. 2024The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bitscited by 308
  10. 2019Universal Approximation with Deep Narrow Networkscited by 200
  11. 2023BitNet: Scaling 1-bit Transformers for Large Language Modelscited by 189
  12. 2018Universality of Deep Convolutional Neural Networkscited by 176
12 more
  1. 2022I-ViT: Integer-only Quantization for Efficient Vision Transformer Inferencecited by 95
  2. 2020Minimum Width for Universal Approximationcited by 94
  3. 2020O(n) Connections are Expressive Enough: Universal Approximability of Sparse Transformerscited by 83
  4. 2023Are Transformers with One Layer Self-Attention Using Low-Rank Weight Matrices Universal Approximators?cited by 47
  5. 2023Sumformer: Universal Approximation for Efficient Transformerscited by 30
  6. 2022F8Net: Fixed-Point 8-bit Only Multiplication for Network Quantizationcited by 28
  7. 2020NITI: Training Integer Neural Networks Using Integer-only Arithmeticcited by 20
  8. 2023Minimum width for universal approximation using ReLU networks on compact domaincited by 18
  9. 2018On the Universal Approximability and Complexity Bounds of Quantized ReLU Neural Networkscited by 12
  10. 2022Approximation speed of quantized vs. unquantized ReLU neural networks and beyondcited by 8
  11. 2024Expressive Power of ReLU and Step Networks under Floating-Point Operationscited by 5
  12. 2021iRNN: Integer-only Recurrent Neural Networkcited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 3.

4 papers.

  1. 2025Quantization vs Pruning: Insights from the Strong Lottery Ticket Hypothesiscited by 1
  2. 2026Expressive Power of Floating-Point Neural Networks with Arbitrary Reduction Orders and Inexact Activation Implementationscited by 1
  3. 2026When Can Depth Replace Precision? A Resource Theory of Quantized Neural Computationcited by 1
  4. 2026A Unified Framework for Quantized and Continuous Strong Lottery Ticketscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Arbitrary-Accuracy Neural Approximation with Optimal Neuron Count and Near-Optimal Bit Complexity
  2. 2023Towards Efficient Verification of Quantized Neural Networks
  3. 2022Few-Bit Backward: Quantized Gradients of Activation Functions for Memory Footprint Reduction
  4. 2021Training Quantized Neural Networks to Global Optimality via Semidefinite Programming
  5. 2025Verifying Quantized Graph Neural Networks is PSPACE-complete
  6. 2026Binary Quantized Neural Network Training Is W[1]-Hard Parameterized by Input and Output Dimensions
  7. 2024Gradient-Free Training of Quantized Neural Networks
  8. 2021Approximation of functions with one-bit neural networks