# Twin Network Augmentation: A Novel Training Strategy for Improved Spiking Neural Networks and Efficient Weight Quantization

arXiv: https://arxiv.org/abs/2409.15849 · HTML: https://mlsanity.com/paper/2409.15849/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 33 of its 52 references that are in the map.
- First posted on arXiv 24 Sep 2024.

## Builds on (most cited first, 30 of 33)

- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [mixup: Beyond Empirical Risk Minimization](https://mlsanity.com/paper/1710.09412/) (2017)
- [Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms](https://mlsanity.com/paper/1708.07747/) (2017)
- [Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification](https://mlsanity.com/paper/1502.01852/) (2015)
- [CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Features](https://mlsanity.com/paper/1905.04899/) (2019)
- [FitNets: Hints for Thin Deep Nets](https://mlsanity.com/paper/1412.6550/) (2014)
- [Knowledge Distillation: A Survey](https://mlsanity.com/paper/2006.05525/) (2020)
- [Random Erasing Data Augmentation](https://mlsanity.com/paper/1708.04896/) (2017)
- [Surrogate Gradient Learning in Spiking Neural Networks](https://mlsanity.com/paper/1901.09948/) (2019)
- [Mixture-of-Experts with Expert Choice Routing](https://mlsanity.com/paper/2202.09368/) (2022)
- [BinaryConnect: Training Deep Neural Networks with binary weights during propagations](https://mlsanity.com/paper/1511.00363/) (2015)
- [Incorporating Learnable Membrane Time Constant to Enhance Learning of Spiking Neural Networks](https://mlsanity.com/paper/2007.05785/) (2020)
- [Spikformer: When Spiking Neural Network Meets Transformer](https://mlsanity.com/paper/2209.15425/) (2022)
- [DropBlock: A regularization method for convolutional networks](https://mlsanity.com/paper/1810.12890/) (2018)
- [Direct Training for Spiking Neural Networks: Faster, Larger, Better](https://mlsanity.com/paper/1809.05793/) (2018)
- [Ternary Weight Networks](https://mlsanity.com/paper/1605.04711/) (2016)
- [Temporal Efficient Training of Spiking Neural Network via Gradient Re-weighting](https://mlsanity.com/paper/2202.11946/) (2022)
- [Efficient Neuromorphic Signal Processing with Loihi 2](https://mlsanity.com/paper/2111.03746/) (2021)
- [SpikeGPT: Generative Pre-trained Language Model with Spiking Neural Networks](https://mlsanity.com/paper/2302.13939/) (2023)
- [Advancing Spiking Neural Networks towards Deep Residual Learning](https://mlsanity.com/paper/2112.08954/) (2021)
- [Comprehensive SNN Compression Using ADMM Optimization and Activity Regularization](https://mlsanity.com/paper/1911.00822/) (2019)
- [Constructing Deep Spiking Neural Networks from Artificial Neural Networks with Knowledge Distillation](https://mlsanity.com/paper/2304.05627/) (2023)
- [SEENN: Towards Temporal Spiking Early-Exit Neural Networks](https://mlsanity.com/paper/2304.01230/) (2023)
- [Exploring the Connection Between Binary and Spiking Neural Networks](https://mlsanity.com/paper/2002.10064/) (2020)
- [TCJA-SNN: Temporal-Channel Joint Attention for Spiking Neural Networks](https://mlsanity.com/paper/2206.10177/) (2022)
- [Efficient Video and Audio processing with Loihi 2](https://mlsanity.com/paper/2310.03251/) (2023)
- [BS4NN: Binarized Spiking Neural Networks with Temporal Coding and Learning](https://mlsanity.com/paper/2007.04039/) (2020)
- [Network Augmentation for Tiny Deep Learning](https://mlsanity.com/paper/2110.08890/) (2021)
- [Co-learning synaptic delays, weights and adaptation in spiking neural networks](https://mlsanity.com/paper/2311.16112/) (2023)

## Built on by (most cited first, 1 of 1)

- [Representation Convergence: Mutual Distillation is Secretly a Form of Regularization](https://mlsanity.com/paper/2501.02481/) (2025)

## Abstract (arXiv)

The proliferation of Artificial Neural Networks (ANNs) has led to increased energy consumption, raising concerns about their sustainability. Spiking Neural Networks (SNNs), which are inspired by biological neural systems and operate using sparse, event-driven spikes to communicate information between neurons, offer a potential solution due to their lower energy requirements. An alternative technique for reducing a neural network's footprint is quantization, which compresses weight representations to decrease memory usage and energy consumption. In this study, we present Twin Network Augmentation (TNA), a novel training framework aimed at improving the performance of SNNs while also facilitating an enhanced compression through low-precision quantization of weights. TNA involves co-training an SNN with a twin network, optimizing both networks to minimize their cross-entropy losses and the mean squared error between their output logits. We demonstrate that TNA significantly enhances classification performance across various vision datasets and in addition is particularly effective when applied when reducing SNNs to ternary weight precision. Notably, during inference , only the ternary SNN is retained, significantly reducing the network in number of neurons, connectivity and weight size representation. Our results show that TNA outperforms traditional knowledge distillation methods and achieves state-of-the-art performance for the evaluated network architecture on benchmark datasets, including CIFAR-10, CIFAR-100, and CIFAR-10-DVS. This paper underscores the effectiveness of TNA in bridging the performance gap between SNNs and ANNs and suggests further exploration into the application of TNA in different network architectures and datasets.