mlsanityData to 7 Oct 2026

Dancing along Battery: Enabling Transformer with Run-time Reconfigurability on Mobile Devices

arXiv 2102.06336 ↗First posted 12 Feb 2021cs.LGcited by 9

Not on the map. Why?

Abstract (arXiv)

A pruning-based AutoML framework for run-time reconfigurability, namely RT3, is proposed in this work. This enables Transformer-based large Natural Language Processing (NLP) models to be efficiently executed on resource-constrained mobile devices and reconfigured (i.e., switching models for dynamic hardware conditions) at run-time. Such reconfigurability is the key to save energy for battery-powered mobile devices, which widely use dynamic voltage and frequency scaling (DVFS) technique for hardware reconfiguration to prolong battery life. In this work, we creatively explore a hybrid block-structured pruning (BP) and pattern pruning (PP) for Transformer-based models and first attempt to combine hardware and software reconfiguration to maximally save energy for battery-powered mobile devices. Specifically, RT3 integrates two-level optimizations: First, it utilizes an efficient BP as the first-step compression for resource-constrained mobile devices; then, RT3 heuristically generates a shrunken search space based on the first level optimization and searches multiple pattern sets with diverse sparsity for PP via reinforcement learning to support lightweight software reconfiguration, which corresponds to available frequency levels of DVFS (i.e., hardware reconfiguration). At run-time, RT3 can switch the lightweight pattern sets within 45ms to guarantee the required real-time constraint at different frequency levels. Results further show that RT3 can prolong battery life over 4x improvement with less than 1% accuracy loss for Transformer and 1.5% score decrease for DistilBERT.

Key facts. Cited by 9 papers in the map (data through 7 Oct 2026). Builds on 23 of its 36 references that are in the map. First posted on arXiv 12 Feb 2021.

Builds on

23 of 36 references.

References · 23

  1. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  2. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  3. 2016Pointer Sentinel Mixture Modelscited by 2,992
  4. 2015Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Codingcited by 2,291
  5. 2015Learning both Weights and Connections for Efficient Neural Networkscited by 1,998
  6. 2016Neural Architecture Search with Reinforcement Learningcited by 1,489
  7. 2016Learning Structured Sparsity in Deep Neural Networkscited by 538
  8. 2019Edge AI: On-Demand Accelerating Deep Neural Network Inference via Edge Computingcited by 150
  9. 2019Adaptive Attention Span in Transformerscited by 148
  10. 2020HAT: Hardware-Aware Transformers for Efficient Natural Language Processingcited by 136
  11. 2020PatDNN: Achieving Real-Time DNN Execution on Mobile Devices with Pattern-based Weight Pruningcited by 75
  12. 2019Hardware/Software Co-Exploration of Neural Architecturescited by 50
11 more
  1. 2019PCONV: The Missing but Desirable Sparsity in DNN Weight Pruning for Real-time Execution on Mobile Devicescited by 48
  2. 2020Co-Exploration of Neural Architectures and Heterogeneous ASIC Accelerator Designs Targeting Multiple Taskscited by 43
  3. 2020Efficient Transformer-based Large Scale Language Representations using Hardware-friendly Block Structured Pruningcited by 39
  4. 2019Accuracy vs. Efficiency: Achieving Both through FPGA-Implementation Aware Neural Architecture Searchcited by 38
  5. 2017Exploring the Regularity of Sparse Structure in Convolutional Neural Networkscited by 37
  6. 2019Device-Circuit-Architecture Co-Exploration for Computing-in-Memory Neural Acceleratorscited by 32
  7. 2020Standing on the Shoulders of Giants: Hardware and Neural Architecture Co-Search with Hot Startcited by 22
  8. 2019On Neural Architecture Search for Resource-Constrained Hardware Platformscited by 18
  9. 2019Achieving Super-Linear Speedup across Multi-FPGA for Real-Time DNN Inferencecited by 13
  10. 2020NASS: Optimizing Secure Inference via Neural Architecture Searchcited by 7
  11. 2020CSB-RNN: A Faster-than-Realtime RNN Acceleration Framework with Compressed Structured Blockscited by 4

Built on by

Cited by year

Citing papers per year: 2021: 3; 2022: 3; 2023: 0; 2024: 2; 2025: 1.

9 papers.

  1. 2022The Larger The Fairer? Small Neural Networks Can Achieve Fairness for Edge Devicescited by 21
  2. 2021Can Noise on Qubits Be Learned in Quantum Neural Network? A Case Study on QuantumFlowcited by 13
  3. 2021Exploration of Quantum Neural Architecture by Mixing Quantum Neuron Designscited by 11
  4. 2022All-in-One: A Highly Representative DNN Pruning Framework for Edge Devices with Dynamic Power Managementcited by 5
  5. 2022EVE: Environmental Adaptive Neural Network Models for Low-power Energy Harvesting Systemcited by 2
  6. 2024A Physics-guided Generative AI Toolkit for Geophysical Monitoringcited by 2
  7. 2024AyE-Edge: Automated Deployment Space Search Empowering Accuracy yet Efficient Real-Time Object Detection on the Edgecited by 1
  8. 2021BSC: Block-based Stochastic Computing to Enable Accurate and Efficient TinyMLcited by 0
  9. 2025CascadedViT: Cascaded Chunk-FeedForward and Cascaded Group Attention Vision Transformercited by 0