# SCoTTi: Save Computation at Training Time with an adaptive framework

arXiv: https://arxiv.org/abs/2312.12483 · HTML: https://mlsanity.com/paper/2312.12483/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 47 references that are in the map.
- First posted on arXiv 19 Dec 2023.

## Builds on (most cited first, 28 of 28)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks](https://mlsanity.com/paper/1803.03635/) (2018)
- [Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation](https://mlsanity.com/paper/1308.3432/) (2013)
- [Learning both Weights and Connections for Efficient Neural Networks](https://mlsanity.com/paper/1506.02626/) (2015)
- [Moment Matching for Multi-Source Domain Adaptation](https://mlsanity.com/paper/1812.01754/) (2018)
- [Pruning Filters for Efficient ConvNets](https://mlsanity.com/paper/1608.08710/) (2016)
- [A White Paper on Neural Network Quantization](https://mlsanity.com/paper/2106.08295/) (2021)
- [The Computational Limits of Deep Learning](https://mlsanity.com/paper/2007.05558/) (2020)
- [A Survey on Distributed Machine Learning](https://mlsanity.com/paper/1912.09789/) (2019)
- [Quantization Networks](https://mlsanity.com/paper/1911.09464/) (2019)
- [A Systematic DNN Weight Pruning Framework using Alternating Direction Method of Multipliers](https://mlsanity.com/paper/1804.03294/) (2018)
- [Online Learning Rate Adaptation with Hypergradient Descent](https://mlsanity.com/paper/1703.04782/) (2017)
- [Provable Filter Pruning for Efficient Neural Networks](https://mlsanity.com/paper/1911.07412/) (2019)
- [SGD Learns Over-parameterized Networks that Provably Generalize on Linearly Separable Data](https://mlsanity.com/paper/1710.10174/) (2017)
- [Revisiting Random Channel Pruning for Neural Network Compression](https://mlsanity.com/paper/2205.05676/) (2022)
- [TransTailor: Pruning the Pre-trained Model for Improved Transfer Learning](https://mlsanity.com/paper/2103.01542/) (2021)
- [Straggler-aware Distributed Learning: Communication Computation Latency Trade-off](https://mlsanity.com/paper/2004.04948/) (2020)
- [To update or not to update? Neurons at equilibrium in deep models](https://mlsanity.com/paper/2207.09455/) (2022)
- [Efficient On-device Training via Gradient Filtering](https://mlsanity.com/paper/2301.00330/) (2023)
- [Privacy-preserving Decentralized Federated Learning over Time-varying Communication Graph](https://mlsanity.com/paper/2210.00325/) (2022)
- [Pruning artificial neural networks: a way to find well-generalizing, high-entropy sharp minima](https://mlsanity.com/paper/2004.14765/) (2020)
- [The rise of the lottery heroes: why zero-shot pruning is hard](https://mlsanity.com/paper/2202.12400/) (2022)
- [Neuron-based Pruning of Deep Neural Networks with Better Generalization using Kronecker Factored Curvature Approximation](https://mlsanity.com/paper/2111.08577/) (2021)
- [A Federated Deep Learning Framework for Privacy Preservation and Communication Efficiency](https://mlsanity.com/paper/2001.09782/) (2020)
- [Weight Pruning via Adaptive Sparsity Loss](https://mlsanity.com/paper/2006.02768/) (2020)
- [Optimized preprocessing and Tiny ML for Attention State Classification](https://mlsanity.com/paper/2303.11371/) (2023)

## Built on by (most cited first, 3 of 3)

- [Study of Training Dynamics for Memory-Constrained Fine-Tuning](https://mlsanity.com/paper/2510.19675/) (2025)
- [Neural Velocity for hyperparameter tuning](https://mlsanity.com/paper/2507.05309/) (2025)
- [Memory Constrained Dynamic Subnetwork Update for Transfer Learning](https://mlsanity.com/paper/2510.20979/) (2025)

## Abstract (arXiv)

On-device training is an emerging approach in machine learning where models are trained on edge devices, aiming to enhance privacy protection and real-time performance. However, edge devices typically possess restricted computational power and resources, making it challenging to perform computationally intensive model training tasks. Consequently, reducing resource consumption during training has become a pressing concern in this field. To this end, we propose SCoTTi (Save Computation at Training Time), an adaptive framework that addresses the aforementioned challenge. It leverages an optimizable threshold parameter to effectively reduce the number of neuron updates during training which corresponds to a decrease in memory and computation footprint. Our proposed approach demonstrates superior performance compared to the state-of-the-art methods regarding computational resource savings on various commonly employed benchmarks and popular architectures, including ResNets, MobileNet, and Swin-T.