mlsanityData to 14 Oct 2026

Extremely Low Bit Neural Network: Squeeze the Last Bit Out with ADMM

arXiv 1707.09870 ↗First posted 24 Jul 2017cs.CVcited by 61

Not on the map. Why?

Abstract (arXiv)

Although deep learning models are highly effective for various learning tasks, their high computational costs prohibit the deployment to scenarios where either memory or computational resources are limited. In this paper, we focus on compressing and accelerating deep models with network weights represented by very small numbers of bits, referred to as extremely low bit neural network. We model this problem as a discretely constrained optimization problem. Borrowing the idea from Alternating Direction Method of Multipliers (ADMM), we decouple the continuous parameters from the discrete constraints of network, and cast the original hard problem into several subproblems. We propose to solve these subproblems using extragradient and iterative quantization algorithms that lead to considerably faster convergency compared to conventional optimization methods. Extensive experiments on image recognition and object detection verify that the proposed algorithm is more effective than state-of-the-art approaches when coming to extremely low bit neural network.

Key facts. Cited by 61 papers in the map (data through 14 Oct 2026). First posted on arXiv 24 Jul 2017.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 21; 2022: 15; 2023: 9; 2024: 5; 2025: 6; 2026: 5.

61 papers.

Sort:

By area:

  1. 2021A Survey of Quantization Methods for Efficient Neural Network Inferencecited by 815
  2. 2021Pruning and Quantization for Deep Neural Network Acceleration: A Surveycited by 263
  3. 2022The case for 4-bit precision: k-bit Inference Scaling Lawscited by 251
  4. 2021ReCU: Reviving the Dead Weights in Binary Neural Networkscited by 52
  5. 2021Dynamic Network Quantization for Efficient Video Inferencecited by 36
  6. 2021QuPeD: Quantized Personalization via Distillation with Applications to Federated Learningcited by 32
  7. 2022Auto-ViT-Acc: An FPGA-Aware Automatic Acceleration Framework for Vision Transformer with Mixed-Scheme Quantizationcited by 29
  8. 2022Perception-Distortion Balanced ADMM Optimization for Single-Image Super-Resolutioncited by 19
  9. 2023Estimator Meets Equilibrium Perspective: A Rectified Straight Through Estimator for Binary Neural Networks Trainingcited by 19
  10. 2021SiMaN: Sign-to-Magnitude Network Binarizationcited by 18
  11. 2023Towards Artificial General Intelligence (AGI) in the Internet of Things (IoT): Opportunities and Challengescited by 14
  12. 2021Demystifying and Generalizing BinaryConnectcited by 11
38 more
  1. 2021GRIM: A General, Real-Time Deep Learning Inference Framework for Mobile Devices based on Fine-Grained Structured Weight Sparsitycited by 10
  2. 2021Mixed Precision of Quantization of Transformer Language Models for Speech Recognitioncited by 10
  3. 2025Synthetic Text Generation for Training Large Language Models via Gradient Matchingcited by 10
  4. 2025Tequila: Trapping-free Ternary Quantization for Large Language Modelscited by 9
  5. 2024Workflow Optimization for Parallel Split Learningcited by 8
  6. 2022Mixed-Precision Neural Networks: A Surveycited by 7
  7. 2021RMSMP: A Novel Deep Neural Network Quantization Framework with Row-wise Mixed Schemes and Multiple Precisionscited by 6
  8. 2023Quantization Aware Factorization for Deep Neural Network Compressioncited by 6
  9. 2021Mixed Precision Low-bit Quantization of Neural Network Language Models for Speech Recognitioncited by 5
  10. 2022Sub-mW Keyword Spotting on an MCU: Analog Binary Feature Extraction and Binary Neural Networkscited by 5
  11. 2026Sherry: Hardware-Efficient 1.25-Bit Ternary Quantization via Fine-grained Sparsificationcited by 5
  12. 2021Elastic Significant Bit Quantization and Acceleration for Deep Neural Networkscited by 4
  13. 2022Deep Neural Networks pruning via the Structured Perspective Regularizationcited by 4
  14. 2022Hyperspherical Quantization: Toward Smaller and More Accurate Modelscited by 4
  15. 2023Learning Discretized Neural Networks under Ricci Flowcited by 4
  16. 2024Quasar-ViT: Hardware-Oriented Quantization-Aware Architecture Search for Vision Transformerscited by 4
  17. 2021Alternate Model Growth and Pruning for Efficient Training of Recommendation Systemscited by 3
  18. 20213U-EdgeAI: Ultra-Low Memory Training, Ultra-Low BitwidthQuantization, and Ultra-Low Latency Accelerationcited by 3
  19. 2021Model compression as constrained optimization, with application to neural nets. Part V: combining compressionscited by 3
  20. 2022Seeking Interpretability and Explainability in Binary Activated Neural Networkscited by 3
  21. 2023Data-Side Efficiencies for Lightweight Convolutional Neural Networkscited by 3
  22. 2021Efficient Visual Recognition with Deep Neural Networks: A Survey on Recent Advances and New Directionscited by 2
  23. 2021Low-bit Quantization of Recurrent Neural Network Language Models Using Alternating Direction Methods of Multiplierscited by 2
  24. 2022Vau da muntanialas: Energy-efficient multi-die scalable acceleration of RNN inferencecited by 2
  25. 2022Distributed Semi-supervised Fuzzy Regression with Interpolation Consistency Regularizationcited by 2
  26. 2022AskewSGD : An Annealed interval-constrained Optimisation method to train Quantized Neural Networkscited by 2
  27. 2023Ternary Quantization: A Surveycited by 2
  28. 2024Differentiable Search for Finding Optimal Quantization Strategycited by 2
  29. 2024Few-Class Arena: A Benchmark for Efficient Selection of Vision Models and Dataset Difficulty Measurementcited by 2
  30. 2025Efficient Continual Learning in Keyword Spotting using Binary Neural Networkscited by 2
  31. 2021CBP: Backpropagation with constraint on weight precision using a pseudo-Lagrange multiplier methodcited by 1
  32. 2022DNN Training Acceleration via Exploring GPGPU Friendly Sparsitycited by 1
  33. 2023Enhancing Quantised End-to-End ASR Models via Personalisationcited by 1
  34. 2025BiVM: Accurate Binarized Neural Network for Efficient Video Mattingcited by 1
  35. 2026NanoQuant: Efficient Sub-1-Bit Quantization of Large Language Modelscited by 1
  36. 2021Efficient Micro-Structured Weight Unification and Pruning for Neural Network Compressioncited by 0
  37. 2021Architecture Aware Latency Constrained Sparse Neural Networkscited by 0
  38. 2021Unified Signal Compression Using a GAN with Iterative Latent Representation Optimizationcited by 0

50 most cited of 61 All 61 (JSON)