mlsanityData to 7 Oct 2026

Google Neural Network Models for Edge Devices: Analyzing and Mitigating Machine Learning Inference Bottlenecks

arXiv 2109.14320 ↗First posted 29 Sep 2021cs.AR cs.LGcited by 22

Not on the map. Why?

Abstract (arXiv)

Emerging edge computing platforms often contain machine learning (ML) accelerators that can accelerate inference for a wide range of neural network (NN) models. These models are designed to fit within the limited area and energy constraints of the edge computing platforms, each targeting various applications (e.g., face detection, speech recognition, translation, image captioning, video analytics). To understand how edge ML accelerators perform, we characterize the performance of a commercial Google Edge TPU, using 24 Google edge NN models (which span a wide range of NN model types) and analyzing each NN layer within each model. We find that the Edge TPU suffers from three major shortcomings: (1) it operates significantly below peak computational throughput, (2) it operates significantly below its theoretical energy efficiency, and (3) its memory system is a large energy and performance bottleneck. Our characterization reveals that the one-size-fits-all, monolithic design of the Edge TPU ignores the high degree of heterogeneity both across different NN models and across different NN layers within the same NN model, leading to the shortcomings we observe. We propose a new acceleration framework called Mensa. Mensa incorporates multiple heterogeneous edge ML accelerators (including both on-chip and near-data accelerators), each of which caters to the characteristics of a particular subset of NN models and layers. During NN inference, for each NN layer, Mensa decides which accelerator to schedule the layer on, taking into account both the optimality of each accelerator for the layer and layer-to-layer communication costs. Averaged across all 24 Google edge NN models, Mensa improves energy efficiency and throughput by 3.0x and 3.1x over the Edge TPU, and by 2.4x and 4.3x over Eyeriss~v2, a state-of-the-art accelerator.

Key facts. Cited by 22 papers in the map (data through 7 Oct 2026). Builds on 45 of its 104 references that are in the map. First posted on arXiv 29 Sep 2021.

Builds on

45 of 104 references.

References · 45

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  3. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  4. 2014Going Deeper with Convolutionscited by 4,231
  5. 2014Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translationcited by 4,016
  6. 2017MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applicationscited by 3,827
  7. 2014Sequence to Sequence Learning with Neural Networkscited by 2,974
  8. 2014Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modelingcited by 2,341
  9. 2013Visualizing and Understanding Convolutional Networkscited by 2,278
  10. 2016Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translationcited by 1,471
  11. 2015Convolutional LSTM Network: A Machine Learning Approach for Precipitation Nowcastingcited by 1,428
  12. 2015Show, Attend and Tell: Neural Image Caption Generation with Visual Attentioncited by 1,421
33 more
  1. 2014On the Properties of Neural Machine Translation: Encoder-Decoder Approachescited by 1,303
  2. 2017ShuffleNet: An Extremely Efficient Convolutional Neural Network for Mobile Devicescited by 1,194
  3. 2018ShuffleNet V2: Practical Guidelines for Efficient CNN Architecture Designcited by 1,132
  4. 2016SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <0.5MB model sizecited by 1,054
  5. 2014Deep Visual-Semantic Alignments for Generating Image Descriptionscited by 966
  6. 2014Show and Tell: A Neural Image Caption Generatorcited by 877
  7. 2017In-Datacenter Performance Analysis of a Tensor Processing Unitcited by 610
  8. 2013Generating Sequences With Recurrent Neural Networkscited by 549
  9. 2014Long-term Recurrent Convolutional Networks for Visual Recognition and Descriptioncited by 483
  10. 2015Unsupervised Learning of Video Representations using LSTMscited by 434
  11. 2015Recent Advances in Convolutional Neural Networkscited by 360
  12. 2015LSTM: A Search Space Odysseycited by 315
  13. 2018Streaming End-to-end Speech Recognition For Mobile Devicescited by 230
  14. 2017SCNN: An Accelerator for Compressed-sparse Convolutional Neural Networkscited by 132
  15. 2014Long Short-Term Memory Based Recurrent Neural Network Architectures for Large Vocabulary Speech Recognitioncited by 111
  16. 2018Exploring Architectures, Data and Units For Streaming End-to-End Speech Recognition with RNN-Transducercited by 95
  17. 2014Video (language) modeling: a baseline for generative models of natural videoscited by 92
  18. 2019Improving RNN Transducer Modeling for End-to-End Speech Recognitioncited by 68
  19. 2016Smart Reply: Automated Response Suggestion for Emailcited by 54
  20. 2016ESE: Efficient Speech Recognition Engine with Sparse LSTM on FPGAcited by 50
  21. 2016Convolutional Recurrent Neural Networks for Music Classificationcited by 49
  22. 2019RecNMP: Accelerating Personalized Recommendation with Near-Memory Processingcited by 49
  23. 2017Convolutional Recurrent Neural Networks for Polyphonic Sound Event Detectioncited by 46
  24. 2018Focus: Querying Large Video Datasets with Low Latency and Low Costcited by 46
  25. 2016Face Detection with the Faster R-CNNcited by 34
  26. 2016Neural Speech Recognizer: Acoustic-to-Word LSTM Model for Large Vocabulary Speech Recognitioncited by 33
  27. 2016Maximizing CNN Accelerator Efficiency Through Resource Partitioningcited by 29
  28. 2019Processing Data Where It Makes Sense: Enabling In-Memory Computationcited by 21
  29. 2020Optimizing Grouped Convolutions on Edge Devicescited by 12
  30. 2019MASR: A Modular Accelerator for Sparse RNNscited by 8
  31. 2017E-PUR: An Energy-Efficient Processing Unit for Recurrent Neural Networkscited by 7
  32. 2019Serving Recurrent Neural Networks Efficiently with a Spatial Acceleratorcited by 2
  33. 2018Enabling the Adoption of Processing-in-Memory: Challenges, Mechanisms, Future Research Directionscited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 10; 2023: 4; 2024: 2; 2025: 3; 2026: 3.

22 papers.

Sort:

  1. 2023High-performance deep spiking neural networks with 0.3 spikes per neuroncited by 22
  2. 2022An Exact Mapping From ReLU Networks to Spiking Neural Networkscited by 12
  3. 2022Special Session: Towards an Agile Design Methodology for Efficient, Reliable, and Secure ML Systemscited by 6
  4. 2023Shared Memory-contention-aware Concurrent DNN Execution for Diversely Heterogeneous System-on-Chipscited by 5
  5. 2024PIM-Opt: Demystifying Distributed Optimization Algorithms on a Real-World Processing-In-Memory Systemcited by 5
  6. 2022Machine Learning Training on a Real Processing-in-Memory Systemcited by 4
  7. 2022Accelerating Neural Network Inference with Processing-in-DRAM: From the Edge to the Cloudcited by 4
  8. 2022An Experimental Evaluation of Machine Learning Training on a Real Processing-in-Memory Systemcited by 2
  9. 2022EcoFlow: Efficient Convolutional Dataflows for Low-Power Neural Network Acceleratorscited by 1
  10. 2022DeepSketch: A New Machine Learning-Based Reference Search Technique for Post-Deduplication Delta Compressioncited by 1
  11. 2022Heterogeneous Data-Centric Architectures for Modern Data-Intensive Applications: Case Studies in Machine Learning and Databasescited by 1
  12. 2022Hardware-aware mobile building block evaluation for computer visioncited by 1
10 more
  1. 2023TransPimLib: A Library for Efficient Transcendental Functions on Processing-in-Memory Systemscited by 1
  2. 2023VEDLIoT -- Next generation accelerated AIoT systems and applicationscited by 1
  3. 2024SwiftRL: Towards Efficient Reinforcement Learning on Real Processing-In-Memory Systemscited by 1
  4. 2025Harmonia: Enhancing Data Placement and Migration in Hybrid Storage Systems via Multi-Agent Reinforcement Learningcited by 1
  5. 2025A Study on Inference Latency for Vision Transformers on Mobile Devicescited by 1
  6. 2022NEON: Enabling Efficient Support for Nonlinear Operations in Resistive RAM-based Neural Network Acceleratorscited by 0
  7. 2025DCC: Data-Centric Compilation of Machine Learning Kernels for Processing-In-Memory Architecturescited by 0
  8. 2026T3C: Test-Time Tensor Compression with Consistency Guaranteescited by 0
  9. 2026Machine Learning-Driven Intelligent Memory System Design: From On-Chip Caches to Storagecited by 0
  10. 2026SISA: A Scale-In Systolic Array for GEMM Accelerationcited by 0