mlsanityData to 14 Oct 2026

SCoTTi: Save Computation at Training Time with an adaptive framework

arXiv 2312.12483 ↗First posted 19 Dec 2023cs.LG cs.AI cs.CVcited by 3

Not on the map. Why?

Abstract (arXiv)

On-device training is an emerging approach in machine learning where models are trained on edge devices, aiming to enhance privacy protection and real-time performance. However, edge devices typically possess restricted computational power and resources, making it challenging to perform computationally intensive model training tasks. Consequently, reducing resource consumption during training has become a pressing concern in this field. To this end, we propose SCoTTi (Save Computation at Training Time), an adaptive framework that addresses the aforementioned challenge. It leverages an optimizable threshold parameter to effectively reduce the number of neuron updates during training which corresponds to a decrease in memory and computation footprint. Our proposed approach demonstrates superior performance compared to the state-of-the-art methods regarding computational resource savings on various commonly employed benchmarks and popular architectures, including ResNets, MobileNet, and Swin-T.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 28 of its 47 references that are in the map. First posted on arXiv 19 Dec 2023.

Builds on

28 of 47 references.

References · 28

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  3. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  4. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,262
  5. 2013Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computationcited by 2,079
  6. 2015Learning both Weights and Connections for Efficient Neural Networkscited by 2,031
  7. 2018Moment Matching for Multi-Source Domain Adaptationcited by 1,331
  8. 2016Pruning Filters for Efficient ConvNetscited by 1,143
  9. 2021A White Paper on Neural Network Quantizationcited by 472
  10. 2020The Computational Limits of Deep Learningcited by 193
  11. 2019A Survey on Distributed Machine Learningcited by 188
  12. 2019Quantization Networkscited by 153
16 more
  1. 2018A Systematic DNN Weight Pruning Framework using Alternating Direction Method of Multiplierscited by 120
  2. 2017Online Learning Rate Adaptation with Hypergradient Descentcited by 118
  3. 2019Provable Filter Pruning for Efficient Neural Networkscited by 98
  4. 2017SGD Learns Over-parameterized Networks that Provably Generalize on Linearly Separable Datacited by 79
  5. 2022Revisiting Random Channel Pruning for Neural Network Compressioncited by 50
  6. 2021TransTailor: Pruning the Pre-trained Model for Improved Transfer Learningcited by 25
  7. 2020Straggler-aware Distributed Learning: Communication Computation Latency Trade-offcited by 15
  8. 2022To update or not to update? Neurons at equilibrium in deep modelscited by 13
  9. 2023Efficient On-device Training via Gradient Filteringcited by 13
  10. 2022Privacy-preserving Decentralized Federated Learning over Time-varying Communication Graphcited by 11
  11. 2020Pruning artificial neural networks: a way to find well-generalizing, high-entropy sharp minimacited by 8
  12. 2022The rise of the lottery heroes: why zero-shot pruning is hardcited by 7
  13. 2021Neuron-based Pruning of Deep Neural Networks with Better Generalization using Kronecker Factored Curvature Approximationcited by 4
  14. 2020A Federated Deep Learning Framework for Privacy Preservation and Communication Efficiencycited by 2
  15. 2020Weight Pruning via Adaptive Sparsity Losscited by 2
  16. 2023Optimized preprocessing and Tiny ML for Attention State Classificationcited by 1

Built on by

Cited by year

Citing papers per year: 2025: 3.

3 papers.

  1. 2025Study of Training Dynamics for Memory-Constrained Fine-Tuningcited by 1
  2. 2025Neural Velocity for hyperparameter tuningcited by 0
  3. 2025Memory Constrained Dynamic Subnetwork Update for Transfer Learningcited by 0