# 0/1 Deep Neural Networks via Block Coordinate Descent

arXiv: https://arxiv.org/abs/2206.09379 · HTML: https://mlsanity.com/paper/2206.09379/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 58 references that are in the map.
- First posted on arXiv 19 Jun 2022.

## Builds on (most cited first, 26 of 26)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms](https://mlsanity.com/paper/1708.07747/) (2017)
- [Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification](https://mlsanity.com/paper/1502.01852/) (2015)
- [Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)](https://mlsanity.com/paper/1511.07289/) (2015)
- [An overview of gradient descent optimization algorithms](https://mlsanity.com/paper/1609.04747/) (2016)
- [On the Convergence of Adam and Beyond](https://mlsanity.com/paper/1904.09237/) (2019)
- [Self-Normalizing Neural Networks](https://mlsanity.com/paper/1706.02515/) (2017)
- [BinaryConnect: Training Deep Neural Networks with binary weights during propagations](https://mlsanity.com/paper/1511.00363/) (2015)
- [Sparsity in Deep Learning: Pruning and growth for efficient inference and training in neural networks](https://mlsanity.com/paper/2102.00554/) (2021)
- [Searching for Activation Functions](https://mlsanity.com/paper/1710.05941/) (2017)
- [Training Binary Neural Networks with Real-to-Binary Convolutions](https://mlsanity.com/paper/2003.11535/) (2020)
- [Training Neural Networks Without Gradients: A Scalable ADMM Approach](https://mlsanity.com/paper/1605.02026/) (2016)
- [Quasi-hyperbolic momentum and Adam for deep learning](https://mlsanity.com/paper/1810.06801/) (2018)
- [A comprehensive review of Binary Neural Network](https://mlsanity.com/paper/2110.06804/) (2021)
- [Global Convergence of Block Coordinate Descent in Deep Learning](https://mlsanity.com/paper/1803.00225/) (2018)
- [Aggregated Momentum: Stability Through Passive Damping](https://mlsanity.com/paper/1804.00325/) (2018)
- [High-Capacity Expert Binary Networks](https://mlsanity.com/paper/2010.03558/) (2020)
- [Projection Convolutional Neural Networks for 1-bit CNNs via Discrete Back Propagation](https://mlsanity.com/paper/1811.12755/) (2018)
- [Convergent Block Coordinate Descent for Training Tikhonov Regularized Deep Neural Networks](https://mlsanity.com/paper/1711.07354/) (2017)
- [Accurate and Compact Convolutional Neural Networks with Trained Binarization](https://mlsanity.com/paper/1909.11366/) (2019)
- [A Proximal Block Coordinate Descent Algorithm for Deep Neural Network Training](https://mlsanity.com/paper/1803.09082/) (2018)
- [Sparsity Meets Robustness: Channel Pruning for the Feynman-Kac Formalism Principled Robust Deep Neural Nets](https://mlsanity.com/paper/2003.00631/) (2020)
- [Distillation Guided Residual Learning for Binary Convolutional Neural Networks](https://mlsanity.com/paper/2007.05223/) (2020)
- [Computing One-bit Compressive Sensing via Double-Sparsity Constrained Optimization](https://mlsanity.com/paper/2101.03599/) (2021)
- [Quadratic Convergence of Smoothing Newton's Method for 0/1 Loss Optimization](https://mlsanity.com/paper/2103.14987/) (2021)
- [Newton method for ℓ0-regularized optimization](https://mlsanity.com/paper/2004.05132/) (2020)

## Built on by (most cited first, 10 of 10)

- [Teaching CLIP to Count to Ten](https://mlsanity.com/paper/2302.12066/) (2023)
- [ChatGPT-Powered Hierarchical Comparisons for Image Classification](https://mlsanity.com/paper/2311.00206/) (2023)
- [KPL: Training-Free Medical Knowledge Mining of Vision-Language Models](https://mlsanity.com/paper/2501.11231/) (2025)
- [Time Series Analysis in Frequency Domain: A Survey of Open Challenges, Opportunities and Benchmarks](https://mlsanity.com/paper/2504.07099/) (2025)
- [Towards On-device Learning on the Edge: Ways to Select Neurons to Update under a Budget Constraint](https://mlsanity.com/paper/2312.05282/) (2023)
- [Activation Map Compression through Tensor Decomposition for Deep Learning](https://mlsanity.com/paper/2411.06346/) (2024)
- [Model Selection with Model Zoo via Graph Learning](https://mlsanity.com/paper/2404.03988/) (2024)
- [Beyond Low-rank Decomposition: A Shortcut Approach for Efficient On-Device Learning](https://mlsanity.com/paper/2505.05086/) (2025)
- [Efficient Resource-Constrained Training of Transformers via Subspace Optimization](https://mlsanity.com/paper/2510.09160/) (2025)
- [Block Coordinate Descent for Neural Networks Provably Finds Global Minima](https://mlsanity.com/paper/2510.22667/) (2025)

## Abstract (arXiv)

The step function is one of the simplest and most natural activation functions for deep neural networks (DNNs). As it counts 1 for positive variables and 0 for others, its intrinsic characteristics (e.g., discontinuity and no viable information of subgradients) impede its development for several decades. Even if there is an impressive body of work on designing DNNs with continuous activation functions that can be deemed as surrogates of the step function, it is still in the possession of some advantageous properties, such as complete robustness to outliers and being capable of attaining the best learning-theoretic guarantee of predictive accuracy. Hence, in this paper, we aim to train DNNs with the step function used as an activation function (dubbed as 0/1 DNNs). We first reformulate 0/1 DNNs as an unconstrained optimization problem and then solve it by a block coordinate descend (BCD) method. Moreover, we acquire closed-form solutions for sub-problems of BCD as well as its convergence properties. Furthermore, we also integrate ℓ2,0-regularization into 0/1 DNN to accelerate the training process and compress the network scale. As a result, the proposed algorithm has a high performance on classifying MNIST and Fashion-MNIST datasets. As a result, the proposed algorithm has a desirable performance on classifying MNIST, FashionMNIST, Cifar10, and Cifar100 datasets.