mlsanityData to 14 Oct 2026

0/1 Deep Neural Networks via Block Coordinate Descent

arXiv 2206.09379 ↗First posted 19 Jun 2022cs.LGcited by 10

Not on the map. Why?

Abstract (arXiv)

The step function is one of the simplest and most natural activation functions for deep neural networks (DNNs). As it counts 1 for positive variables and 0 for others, its intrinsic characteristics (e.g., discontinuity and no viable information of subgradients) impede its development for several decades. Even if there is an impressive body of work on designing DNNs with continuous activation functions that can be deemed as surrogates of the step function, it is still in the possession of some advantageous properties, such as complete robustness to outliers and being capable of attaining the best learning-theoretic guarantee of predictive accuracy. Hence, in this paper, we aim to train DNNs with the step function used as an activation function (dubbed as 0/1 DNNs). We first reformulate 0/1 DNNs as an unconstrained optimization problem and then solve it by a block coordinate descend (BCD) method. Moreover, we acquire closed-form solutions for sub-problems of BCD as well as its convergence properties. Furthermore, we also integrate ℓ2,0-regularization into 0/1 DNN to accelerate the training process and compress the network scale. As a result, the proposed algorithm has a high performance on classifying MNIST and Fashion-MNIST datasets. As a result, the proposed algorithm has a desirable performance on classifying MNIST, FashionMNIST, Cifar10, and Cifar100 datasets.

Key facts. Cited by 10 papers in the map (data through 14 Oct 2026). Builds on 26 of its 58 references that are in the map. First posted on arXiv 19 Jun 2022.

Builds on

26 of 58 references.

References · 26

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  2. 2017Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithmscited by 4,349
  3. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,611
  4. 2015Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)cited by 1,070
  5. 2016An overview of gradient descent optimization algorithmscited by 1,033
  6. 2019On the Convergence of Adam and Beyondcited by 841
  7. 2017Self-Normalizing Neural Networkscited by 598
  8. 2015BinaryConnect: Training Deep Neural Networks with binary weights during propagationscited by 592
  9. 2021Sparsity in Deep Learning: Pruning and growth for efficient inference and training in neural networkscited by 527
  10. 2017Searching for Activation Functionscited by 169
  11. 2020Training Binary Neural Networks with Real-to-Binary Convolutionscited by 112
  12. 2016Training Neural Networks Without Gradients: A Scalable ADMM Approachcited by 61
14 more
  1. 2018Quasi-hyperbolic momentum and Adam for deep learningcited by 48
  2. 2021A comprehensive review of Binary Neural Networkcited by 45
  3. 2018Global Convergence of Block Coordinate Descent in Deep Learningcited by 38
  4. 2018Aggregated Momentum: Stability Through Passive Dampingcited by 36
  5. 2020High-Capacity Expert Binary Networkscited by 34
  6. 2018Projection Convolutional Neural Networks for 1-bit CNNs via Discrete Back Propagationcited by 29
  7. 2017Convergent Block Coordinate Descent for Training Tikhonov Regularized Deep Neural Networkscited by 26
  8. 2019Accurate and Compact Convolutional Neural Networks with Trained Binarizationcited by 20
  9. 2018A Proximal Block Coordinate Descent Algorithm for Deep Neural Network Trainingcited by 15
  10. 2020Sparsity Meets Robustness: Channel Pruning for the Feynman-Kac Formalism Principled Robust Deep Neural Netscited by 8
  11. 2020Distillation Guided Residual Learning for Binary Convolutional Neural Networkscited by 6
  12. 2021Computing One-bit Compressive Sensing via Double-Sparsity Constrained Optimizationcited by 3
  13. 2021Quadratic Convergence of Smoothing Newton's Method for 0/1 Loss Optimizationcited by 3
  14. 2020Newton method for ℓ0-regularized optimizationcited by 2

Built on by

Cited by year

Citing papers per year: 2023: 3; 2024: 2; 2025: 5.

10 papers.

  1. 2023Teaching CLIP to Count to Tencited by 198
  2. 2023ChatGPT-Powered Hierarchical Comparisons for Image Classificationcited by 22
  3. 2025KPL: Training-Free Medical Knowledge Mining of Vision-Language Modelscited by 10
  4. 2025Time Series Analysis in Frequency Domain: A Survey of Open Challenges, Opportunities and Benchmarkscited by 10
  5. 2023Towards On-device Learning on the Edge: Ways to Select Neurons to Update under a Budget Constraintcited by 6
  6. 2024Activation Map Compression through Tensor Decomposition for Deep Learningcited by 6
  7. 2024Model Selection with Model Zoo via Graph Learningcited by 5
  8. 2025Beyond Low-rank Decomposition: A Shortcut Approach for Efficient On-Device Learningcited by 5
  9. 2025Efficient Resource-Constrained Training of Transformers via Subspace Optimizationcited by 0
  10. 2025Block Coordinate Descent for Neural Networks Provably Finds Global Minimacited by 0