# The High-Dimensional Geometry of Binary Neural Networks

arXiv: https://arxiv.org/abs/1705.07199 · HTML: https://mlsanity.com/paper/1705.07199/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 19 May 2017.

## Built on by (most cited first, 22 of 22)

- [A Survey on Hyperdimensional Computing aka Vector Symbolic Architectures, Part II: Applications, Cognitive Models, and Challenges](https://mlsanity.com/paper/2112.15424/) (2021)
- [AdaBin: Improving Binary Neural Networks with Adaptive Binary Sets](https://mlsanity.com/paper/2208.08084/) (2022)
- [SiMaN: Sign-to-Magnitude Network Binarization](https://mlsanity.com/paper/2102.07981/) (2021)
- [Binary Complex Neural Network Acceleration on FPGA](https://mlsanity.com/paper/2108.04811/) (2021)
- [Demystifying and Generalizing BinaryConnect](https://mlsanity.com/paper/2110.13220/) (2021)
- [A quantitative analysis of semantic information in deep representations of text and images](https://mlsanity.com/paper/2505.17101/) (2025)
- [A Bop and Beyond: A Second Order Optimizer for Binarized Neural Networks](https://mlsanity.com/paper/2104.05124/) (2021)
- [BCNN: Binary Complex Neural Network](https://mlsanity.com/paper/2104.10044/) (2021)
- [Input Layer Binarization with Bit-Plane Encoding](https://mlsanity.com/paper/2305.02885/) (2023)
- [AskewSGD : An Annealed interval-constrained Optimisation method to train Quantized Neural Networks](https://mlsanity.com/paper/2211.03741/) (2022)
- [Straight-Through meets Sparse Recovery: the Support Exploration Algorithm](https://mlsanity.com/paper/2301.13584/) (2023)
- [Unsupervised detection of semantic correlations in big data](https://mlsanity.com/paper/2411.02126/) (2024)
- [A Configurable BNN ASIC using a Network of Programmable Threshold Logic Standard Cells](https://mlsanity.com/paper/2104.01699/) (2021)
- [GAAF: Searching Activation Functions for Binary Neural Networks through Genetic Algorithm](https://mlsanity.com/paper/2206.03291/) (2022)
- [Learning Using a Single Forward Pass](https://mlsanity.com/paper/2402.09769/) (2024)
- [Solving hybrid machine learning tasks by traversing weight space geodesics](https://mlsanity.com/paper/2106.02793/) (2021)
- [Mixture-of-Rookies: Saving DNN Computations by Predicting ReLU Outputs](https://mlsanity.com/paper/2202.04990/) (2022)
- [Saving RNN Computations with a Neuron-Level Fuzzy Memoization Scheme](https://mlsanity.com/paper/2202.06563/) (2022)
- [A Data-Free Analytical Quantization Scheme for Deep Learning Models](https://mlsanity.com/paper/2412.07391/) (2024)
- [Zeros can be Informative: Masked Binary U-Net for Image Segmentation on Tensor Cores](https://mlsanity.com/paper/2601.11660/) (2026)
- [On the Expressive Power of Weight Quantization in Large Language Models](https://mlsanity.com/paper/2606.22249/) (2026)
- [VAD to the Bone: Ultra-Tiny Speech Activity Detection for Edge Deployment](https://mlsanity.com/paper/2607.25870/) (2026)

## Abstract (arXiv)

Recent research has shown that one can train a neural network with binary weights and activations at train time by augmenting the weights with a high-precision continuous latent variable that accumulates small changes from stochastic gradient descent. However, there is a dearth of theoretical analysis to explain why we can effectively capture the features in our data with binary weights and activations. Our main result is that the neural networks with binary weights and activations trained using the method of Courbariaux, Hubara et al. (2016) work because of the high-dimensional geometry of binary vectors. In particular, the ideal continuous vectors that extract out features in the intermediate representations of these BNNs are well-approximated by binary vectors in the sense that dot products are approximately preserved. Compared to previous research that demonstrated the viability of such BNNs, our work explains why these BNNs work in terms of the HD geometry. Our theory serves as a foundation for understanding not only BNNs but a variety of methods that seek to compress traditional neural networks. Furthermore, a better understanding of multilayer binary neural networks serves as a starting point for generalizing BNNs to other neural network architectures such as recurrent neural networks.