# On Expressive Power of Quantized Neural Networks under Fixed-Point Arithmetic

arXiv: https://arxiv.org/abs/2409.00297 · HTML: https://mlsanity.com/paper/2409.00297/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 39 references that are in the map.
- First posted on arXiv 30 Aug 2024.

## Builds on (most cited first, 24 of 24)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [MobileNetV2: Inverted Residuals and Linear Bottlenecks](https://mlsanity.com/paper/1801.04381/) (2018)
- [DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter](https://mlsanity.com/paper/1910.01108/) (2019)
- [Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference](https://mlsanity.com/paper/1712.05877/) (2017)
- [Error bounds for approximations with deep ReLU networks](https://mlsanity.com/paper/1610.01145/) (2016)
- [SpinQuant: LLM quantization with learned rotations](https://mlsanity.com/paper/2405.16406/) (2024)
- [Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activations](https://mlsanity.com/paper/1609.07061/) (2016)
- [Are Transformers universal approximators of sequence-to-sequence functions?](https://mlsanity.com/paper/1912.10077/) (2019)
- [The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bits](https://mlsanity.com/paper/2402.17764/) (2024)
- [Universal Approximation with Deep Narrow Networks](https://mlsanity.com/paper/1905.08539/) (2019)
- [BitNet: Scaling 1-bit Transformers for Large Language Models](https://mlsanity.com/paper/2310.11453/) (2023)
- [Universality of Deep Convolutional Neural Networks](https://mlsanity.com/paper/1805.10769/) (2018)
- [I-ViT: Integer-only Quantization for Efficient Vision Transformer Inference](https://mlsanity.com/paper/2207.01405/) (2022)
- [Minimum Width for Universal Approximation](https://mlsanity.com/paper/2006.08859/) (2020)
- [O(n) Connections are Expressive Enough: Universal Approximability of Sparse Transformers](https://mlsanity.com/paper/2006.04862/) (2020)
- [Are Transformers with One Layer Self-Attention Using Low-Rank Weight Matrices Universal Approximators?](https://mlsanity.com/paper/2307.14023/) (2023)
- [Sumformer: Universal Approximation for Efficient Transformers](https://mlsanity.com/paper/2307.02301/) (2023)
- [F8Net: Fixed-Point 8-bit Only Multiplication for Network Quantization](https://mlsanity.com/paper/2202.05239/) (2022)
- [NITI: Training Integer Neural Networks Using Integer-only Arithmetic](https://mlsanity.com/paper/2009.13108/) (2020)
- [Minimum width for universal approximation using ReLU networks on compact domain](https://mlsanity.com/paper/2309.10402/) (2023)
- [On the Universal Approximability and Complexity Bounds of Quantized ReLU Neural Networks](https://mlsanity.com/paper/1802.03646/) (2018)
- [Approximation speed of quantized vs. unquantized ReLU neural networks and beyond](https://mlsanity.com/paper/2205.11874/) (2022)
- [Expressive Power of ReLU and Step Networks under Floating-Point Operations](https://mlsanity.com/paper/2401.15121/) (2024)
- [iRNN: Integer-only Recurrent Neural Network](https://mlsanity.com/paper/2109.09828/) (2021)

## Built on by (most cited first, 4 of 4)

- [Quantization vs Pruning: Insights from the Strong Lottery Ticket Hypothesis](https://mlsanity.com/paper/2508.11020/) (2025)
- [Expressive Power of Floating-Point Neural Networks with Arbitrary Reduction Orders and Inexact Activation Implementations](https://mlsanity.com/paper/2605.28704/) (2026)
- [When Can Depth Replace Precision? A Resource Theory of Quantized Neural Computation](https://mlsanity.com/paper/2607.23390/) (2026)
- [A Unified Framework for Quantized and Continuous Strong Lottery Tickets](https://mlsanity.com/paper/2607.03860/) (2026)

## Abstract (arXiv)

Existing works on the expressive power of neural networks typically assume real parameters and exact operations. In this work, we study the expressive power of quantized networks under discrete fixed-point parameters and inexact fixed-point operations with round-off errors. We first provide a necessary condition and a sufficient condition on fixed-point arithmetic and activation functions for quantized networks to represent all fixed-point functions from fixed-point vectors to fixed-point numbers. Then, we show that various popular activation functions satisfy our sufficient condition, e.g., Sigmoid, ReLU, ELU, SoftPlus, SiLU, Mish, and GELU. In other words, networks using those activation functions are capable of representing all fixed-point functions. We further show that our necessary condition and sufficient condition coincide under a mild condition on activation functions: e.g., for an activation function σ, there exists a fixed-point number x such that σ(x)=0. Namely, we find a necessary and sufficient condition for a large class of activation functions. We lastly show that even quantized networks using binary weights in -1,1 can also represent all fixed-point functions for practical activation functions.