# Differentiable Weightless Neural Networks

arXiv: https://arxiv.org/abs/2410.11112 · HTML: https://mlsanity.com/paper/2410.11112/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 50 references that are in the map.
- First posted on arXiv 14 Oct 2024.

## Builds on (most cited first, 21 of 21)

- [XGBoost: A Scalable Tree Boosting System](https://mlsanity.com/paper/1603.02754/) (2016)
- [Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms](https://mlsanity.com/paper/1708.07747/) (2017)
- [Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation](https://mlsanity.com/paper/1308.3432/) (2013)
- [Speech Commands: A Dataset for Limited-Vocabulary Speech Recognition](https://mlsanity.com/paper/1804.03209/) (2018)
- [AutoGluon-Tabular: Robust and Accurate AutoML for Structured Data](https://mlsanity.com/paper/2003.06505/) (2020)
- [Learning to Prune Deep Neural Networks via Layer-wise Optimal Brain Surgeon](https://mlsanity.com/paper/1705.07565/) (2017)
- [Training Neural Networks with Fixed Sparse Masks](https://mlsanity.com/paper/2111.09839/) (2021)
- [Scalable Methods for 8-bit Training of Neural Networks](https://mlsanity.com/paper/1805.11046/) (2018)
- [Fast inference of deep neural networks in FPGAs for particle physics](https://mlsanity.com/paper/1804.06913/) (2018)
- [MLPerf Tiny Benchmark](https://mlsanity.com/paper/2106.07597/) (2021)
- [AdderNet: Do We Really Need Multiplications in Deep Learning?](https://mlsanity.com/paper/1912.13200/) (2019)
- [Deep Differentiable Logic Gate Networks](https://mlsanity.com/paper/2210.08277/) (2022)
- [LogicNets: Co-Designed Neural Networks and Circuits for Extreme-Throughput Applications](https://mlsanity.com/paper/2004.03021/) (2020)
- [DeepShift: Towards Multiplication-Less Neural Networks](https://mlsanity.com/paper/1905.13298/) (2019)
- [Adaptive Gradient Quantization for Data-Parallel SGD](https://mlsanity.com/paper/2010.12460/) (2020)
- [ToyADMOS: A Dataset of Miniature-Machine Operating Sounds for Anomalous Sound Detection](https://mlsanity.com/paper/1908.03299/) (2019)
- [PolyLUT: Learning Piecewise Polynomials for Ultra-Low Latency FPGA LUT-based Inference](https://mlsanity.com/paper/2309.02334/) (2023)
- [NeuraLUT: Hiding Neural Network Density in Boolean Synthesizable Functions](https://mlsanity.com/paper/2403.00849/) (2024)
- [Joint Binary Neural Network for Multi-label Learning with Applications to Emotion Classification](https://mlsanity.com/paper/1802.00891/) (2018)
- [ULEEN: A Novel Architecture for Ultra Low-Energy Edge Neural Networks](https://mlsanity.com/paper/2304.10618/) (2023)
- [BiFSMN: Binary Neural Network for Keyword Spotting](https://mlsanity.com/paper/2202.06483/) (2022)

## Built on by (most cited first, 28 of 28)

- [Convolutional Differentiable Logic Gate Networks](https://mlsanity.com/paper/2411.04732/) (2024)
- [NeuraLUT-Assemble: Hardware-aware Assembling of Sub-Neural Networks for Efficient LUT Inference](https://mlsanity.com/paper/2504.00592/) (2025)
- [Light Differentiable Logic Gate Networks](https://mlsanity.com/paper/2510.03250/) (2025)
- [WARP-LUTs -- Walsh-Assisted Relaxation for Probabilistic Look Up Tables](https://mlsanity.com/paper/2510.15655/) (2025)
- [LILogic Net: Compact Logic Gate Networks with Learnable Connectivity for Efficient Hardware Deployment](https://mlsanity.com/paper/2511.12340/) (2025)
- [Scalable Interconnect Learning in Boolean Networks](https://mlsanity.com/paper/2507.02585/) (2025)
- [LL-ViT: Edge Deployable Vision Transformers with Look Up Table Neurons](https://mlsanity.com/paper/2511.00812/) (2025)
- [WARP Logic Neural Networks](https://mlsanity.com/paper/2602.03527/) (2026)
- [AI Compute Architecture and Evolution Trends](https://mlsanity.com/paper/2508.21394/) (2025)
- [Differentiable Weightless Controllers: Learning Logic Circuits for Continuous Control](https://mlsanity.com/paper/2512.01467/) (2025)
- [Inter-patient ECG Arrhythmia Classification with LGNs and LUTNs](https://mlsanity.com/paper/2601.11433/) (2026)
- [Continuous-Flow Data-Rate-Aware CNN Inference on FPGA](https://mlsanity.com/paper/2601.19940/) (2026)
- [Learning Compact Boolean Networks](https://mlsanity.com/paper/2602.05830/) (2026)
- [BitLogic: Training Framework for Gradient-Based FPGA-Native Neural Networks](https://mlsanity.com/paper/2602.07400/) (2026)
- [HGQ-LUT: Fast LUT-Aware Training and Efficient Architectures for DNN Inference](https://mlsanity.com/paper/2604.22293/) (2026)
- [FPGN: Redefining Ultra-Fast Programmable Gate-based Neural Acceleration with Differentiable LUTs](https://mlsanity.com/paper/2607.08427/) (2026)
- [Shrinking the Giant : Quasi-Weightless Transformers for Low Energy Inference](https://mlsanity.com/paper/2411.01818/) (2024)
- [Quantization Effects of Artificial Neural Networks for Embedded Edge-Computing Applications](https://mlsanity.com/paper/2511.05479/) (2025)
- [BEP: A Binary Error Propagation Algorithm for Binary Neural Networks Training](https://mlsanity.com/paper/2512.04189/) (2025)
- [Enhancing LUT-based Deep Neural Networks Inference through Architecture and Connectivity Optimization](https://mlsanity.com/paper/2601.09773/) (2026)
- [TT-Sparse: Learning Sparse Rule Models with Differentiable Truth Tables](https://mlsanity.com/paper/2603.07606/) (2026)
- [GLANCE: Gaze-Led Attention Network for Compressed Edge-inference](https://mlsanity.com/paper/2603.15717/) (2026)
- [From Arithmetic to Logic: The Resilience of Logic and Lookup-Based Neural Networks Under Parameter Bit-Flips](https://mlsanity.com/paper/2603.22770/) (2026)
- [Fitting Multilinear Polynomials for Logic Gate Networks](https://mlsanity.com/paper/2605.08657/) (2026)
- [CascadeLUT: Information-Ordered Streaming Inference for Bandwidth-Constrained FPGAs](https://mlsanity.com/paper/2608.00720/) (2026)
- [DiffLUT-Net: Differentiable Training of FPGA LUT Networks with Learnable Connectivity](https://mlsanity.com/paper/2609.09254/) (2026)
- [Task-Aware Discretization of Differentiable Logic Gate Networks](https://mlsanity.com/paper/2609.33747/) (2026)
- [Overcoming Kernel Redundancy for Scaling Logic Gate Networks](https://mlsanity.com/paper/2610.01069/) (2026)

## Abstract (arXiv)

We introduce the Differentiable Weightless Neural Network (DWN), a model based on interconnected lookup tables. Training of DWNs is enabled by a novel Extended Finite Difference technique for approximate differentiation of binary values. We propose Learnable Mapping, Learnable Reduction, and Spectral Regularization to further improve the accuracy and efficiency of these models. We evaluate DWNs in three edge computing contexts: (1) an FPGA-based hardware accelerator, where they demonstrate superior latency, throughput, energy efficiency, and model area compared to state-of-the-art solutions, (2) a low-power microcontroller, where they achieve preferable accuracy to XGBoost while subject to stringent memory constraints, and (3) ultra-low-cost chips, where they consistently outperform small models in both accuracy and projected hardware area. DWNs also compare favorably against leading approaches for tabular datasets, with higher average rank. Overall, our work positions DWNs as a pioneering solution for edge-compatible high-throughput neural networks.