# BILLNET: A Binarized Conv3D-LSTM Network with Logic-gated residual architecture for hardware-efficient video inference

arXiv: https://arxiv.org/abs/2501.14495 · HTML: https://mlsanity.com/paper/2501.14495/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 38 references that are in the map.
- First posted on arXiv 24 Jan 2025.

## Builds on (most cited first, 30 of 30)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [Squeeze-and-Excitation Networks](https://mlsanity.com/paper/1709.01507/) (2017)
- [MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications](https://mlsanity.com/paper/1704.04861/) (2017)
- [Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset](https://mlsanity.com/paper/1705.07750/) (2017)
- [Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation](https://mlsanity.com/paper/1308.3432/) (2013)
- [ShuffleNet: An Extremely Efficient Convolutional Neural Network for Mobile Devices](https://mlsanity.com/paper/1707.01083/) (2017)
- [A Closer Look at Spatiotemporal Convolutions for Action Recognition](https://mlsanity.com/paper/1711.11248/) (2017)
- [TSM: Temporal Shift Module for Efficient Video Understanding](https://mlsanity.com/paper/1811.08383/) (2018)
- [Learned Step Size Quantization](https://mlsanity.com/paper/1902.08153/) (2019)
- [Can Spatiotemporal 3D CNNs Retrace the History of 2D CNNs and ImageNet?](https://mlsanity.com/paper/1711.09577/) (2017)
- [Rethinking Spatiotemporal Feature Learning: Speed-Accuracy Trade-offs in Video Classification](https://mlsanity.com/paper/1712.04851/) (2017)
- [Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activations](https://mlsanity.com/paper/1609.07061/) (2016)
- [Learning Spatio-Temporal Representation with Pseudo-3D Residual Networks](https://mlsanity.com/paper/1711.10305/) (2017)
- [Residual Attention Network for Image Classification](https://mlsanity.com/paper/1704.06904/) (2017)
- [YouTube-8M: A Large-Scale Video Classification Benchmark](https://mlsanity.com/paper/1609.08675/) (2016)
- [Temporal Segment Networks for Action Recognition in Videos](https://mlsanity.com/paper/1705.02953/) (2017)
- [Ternary Weight Networks](https://mlsanity.com/paper/1605.04711/) (2016)
- [Learning Spatio-Temporal Features with 3D Residual Networks for Action Recognition](https://mlsanity.com/paper/1708.07632/) (2017)
- [Gate-Shift Networks for Video Action Recognition](https://mlsanity.com/paper/1912.00381/) (2019)
- [Resource Efficient 3D Convolutional Neural Networks](https://mlsanity.com/paper/1904.02422/) (2019)
- [Differentiable Joint Pruning and Quantization for Hardware Efficiency](https://mlsanity.com/paper/2007.10463/) (2020)
- [Dynamic Network Quantization for Efficient Video Inference](https://mlsanity.com/paper/2108.10394/) (2021)
- [Tiny Video Networks](https://mlsanity.com/paper/1910.06961/) (2019)
- [Gating Revisited: Deep Multi-layer RNNs That Can Be Trained](https://mlsanity.com/paper/1911.11033/) (2019)
- [Recurrent Neural Networks With Limited Numerical Precision](https://mlsanity.com/paper/1608.06902/) (2016)
- [Effective Quantization Approaches for Recurrent Neural Networks](https://mlsanity.com/paper/1802.02615/) (2018)
- [Revisiting 3D ResNets for Video Recognition](https://mlsanity.com/paper/2109.01696/) (2021)
- [4-bit Quantization of LSTM-based Speech Recognition Models](https://mlsanity.com/paper/2108.12074/) (2021)

## Built on by (most cited first, 1 of 1)

- [Towards Experience Replay for Class-Incremental Learning in Fully-Binary Networks](https://mlsanity.com/paper/2503.07107/) (2025)

## Abstract (arXiv)

Long Short-Term Memory (LSTM) and 3D convolution (Conv3D) show impressive results for many video-based applications but require large memory and intensive computing. Motivated by recent works on hardware-algorithmic co-design towards efficient inference, we propose a compact binarized Conv3D-LSTM model architecture called BILLNET, compatible with a highly resource-constrained hardware. Firstly, BILLNET proposes to factorize the costly standard Conv3D by two pointwise convolutions with a grouped convolution in-between. Secondly, BILLNET enables binarized weights and activations via a MUX-OR-gated residual architecture. Finally, to efficiently train BILLNET, we propose a multi-stage training strategy enabling to fully quantize LSTM layers. Results on Jester dataset show that our method can obtain high accuracy with extremely low memory and computational budgets compared to existing Conv3D resource-efficient models.