mlsanityData to 14 Oct 2026

TIPS: Topologically Important Path Sampling for Anytime Neural Networks

arXiv 2305.08021 ↗First posted 13 May 2023cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Anytime neural networks (AnytimeNNs) are a promising solution to adaptively adjust the model complexity at runtime under various hardware resource constraints. However, the manually-designed AnytimeNNs are biased by designers' prior experience and thus provide sub-optimal solutions. To address the limitations of existing hand-crafted approaches, we first model the training process of AnytimeNNs as a discrete-time Markov chain (DTMC) and use it to identify the paths that contribute the most to the training of AnytimeNNs. Based on this new DTMC-based analysis, we further propose TIPS, a framework to automatically design AnytimeNNs under various hardware constraints. Our experimental results show that TIPS can improve the convergence rate and test accuracy of AnytimeNNs. Compared to the existing AnytimeNNs approaches, TIPS improves the accuracy by 2%-6.6% on multiple datasets and achieves SOTA accuracy-FLOPs tradeoffs.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 41 of its 61 references that are in the map. First posted on arXiv 13 May 2023.

Builds on

41 of 61 references.

References · 41

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  4. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  5. 2018MobileNetV2: Inverted Residuals and Linear Bottleneckscited by 4,795
  6. 2015Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Codingcited by 2,329
  7. 2015Learning both Weights and Connections for Efficient Neural Networkscited by 2,031
  8. 2018DARTS: Differentiable Architecture Searchcited by 1,798
  9. 2016Neural Architecture Search with Reinforcement Learningcited by 1,499
  10. 2019Once-for-All: Train One Network and Specialize it for Efficient Deploymentcited by 787
  11. 2013Exact solutions to the nonlinear dynamics of learning in deep linear neural networkscited by 763
  12. 2017ThiNet: A Filter Level Pruning Method for Deep Neural Network Compressioncited by 442
29 more
  1. 2017SkipNet: Learning Dynamic Routing in Convolutional Networkscited by 332
  2. 2018Slimmable Neural Networkscited by 305
  3. 2019Universally Slimmable Networks and Improved Training Techniquescited by 221
  4. 2016FractalNet: Ultra-Deep Neural Networks without Residualscited by 216
  5. 2020Binary Neural Networks: A Surveycited by 190
  6. 2017BlockDrop: Dynamic Inference Paths in Residual Networkscited by 189
  7. 2018Dynamical Isometry and a Mean Field Theory of CNNs: How to Train 10,000-Layer Vanilla Convolutional Neural Networkscited by 176
  8. 2015Conditional Computation in Neural Networks for faster modelscited by 167
  9. 2017Adaptive Neural Networks for Efficient Inferencecited by 163
  10. 2017Convolutional Networks with Adaptive Inference Graphscited by 148
  11. 2018NetAdapt: Platform-Aware Neural Network Adaptation for Mobile Applicationscited by 147
  12. 2018Dynamic Channel Pruning: Feature Boosting and Suppressioncited by 129
  13. 2019A Signal Propagation Perspective for Pruning Neural Networks at Initializationcited by 96
  14. 2019Single-Path NAS: Designing Hardware-Efficient ConvNets in less than 4 Hourscited by 87
  15. 2021Dynamic Slimmable Networkcited by 83
  16. 2018Channel Gating Neural Networkscited by 80
  17. 2019Improved Techniques for Training Adaptive Deep Networkscited by 66
  18. 2023ZiCo: Zero-shot NAS via Inverse Coefficient of Variation on Gradientscited by 58
  19. 2018You Look Twice: GaterNet for Dynamic Filter Selection in CNNscited by 46
  20. 2021Manifold Regularized Dynamic Network Pruningcited by 43
  21. 2019Batch-Shaping for Learning Conditional Channel Gated Networkscited by 42
  22. 2020EDD: Efficient Differentiable DNN Architecture and Implementation Co-search for Embedded AI Solutionscited by 36
  23. 2019Dual Dynamic Inference: Enabling More Efficient, Adaptive and Controllable Deep Inferencecited by 31
  24. 2019S2DNAS:Transforming Static CNN Model for Dynamic Inference via Neural Architecture Searchcited by 14
  25. 2021FLASH: Fast Neural Architecture Search with Hardware Optimizationcited by 11
  26. 2018Anytime Neural Prediction via Slicing Networks Verticallycited by 7
  27. 2020Joslim: Joint Widths and Weights Optimization for Slimmable Neural Networkscited by 4
  28. 2022Deep Architecture Connectivity Matters for Its Convergence: A Fine-Grained Analysiscited by 4
  29. 2019Supracentrality Analysis of Temporal Networks with Directed Interlayer Couplingcited by 1

Built on by

Not cited yet.