mlsanityData to 14 Oct 2026

SparseByteNN: A Novel Mobile Inference Acceleration Framework Based on Fine-Grained Group Sparsity

arXiv 2310.19509 ↗First posted 30 Oct 2023cs.AI cs.CVcited by 0

On the map: Efficient LLM inference and serving → Early exit and edge inference

Abstract (arXiv)

To address the challenge of increasing network size, researchers have developed sparse models through network pruning. However, maintaining model accuracy while achieving significant speedups on general computing devices remains an open problem. In this paper, we present a novel mobile inference acceleration framework SparseByteNN, which leverages fine-grained kernel sparsity to achieve real-time execution as well as high accuracy. Our framework consists of two parts: (a) A fine-grained kernel sparsity schema with a sparsity granularity between structured pruning and unstructured pruning. It designs multiple sparse patterns for different operators. Combined with our proposed whole network rearrangement strategy, the schema achieves a high compression rate and high precision at the same time. (b) Inference engine co-optimized with the sparse pattern. The conventional wisdom is that this reduction in theoretical FLOPs does not translate into real-world efficiency gains. We aim to correct this misconception by introducing a family of efficient sparse kernels for ARM and WebAssembly. Equipped with our efficient implementation of sparse primitives, we show that sparse versions of MobileNet-v1 outperform strong dense baselines on the efficiency-accuracy curve. Experimental results on Qualcomm 855 show that for 30% sparse MobileNet-v1, SparseByteNN achieves 1.27x speedup over the dense version and 1.29x speedup over the state-of-the-art sparse inference engine MNN with a slight accuracy drop of 0.224%. The source code of SparseByteNN will be available at https://github.com/lswzjuer/SparseByteNN

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Early exit and edge inference, one of 12 directions in Efficient LLM inference and serving. Builds on 26 of its 38 references that are in the map. First posted on arXiv 30 Oct 2023.

Builds on

26 of 38 references.

References · 26

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,364
  3. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,442
  4. 2018MobileNetV2: Inverted Residuals and Linear Bottleneckscited by 4,795
  5. 2017MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applicationscited by 3,845
  6. 2015Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Codingcited by 2,329
  7. 2015Learning both Weights and Connections for Efficient Neural Networkscited by 2,031
  8. 2016Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learningcited by 1,512
  9. 2016Pruning Filters for Efficient ConvNetscited by 1,143
  10. 2017Learning Efficient Convolutional Networks through Network Slimmingcited by 676
  11. 2017To prune, or not to prune: exploring the efficacy of pruning for model compressioncited by 603
  12. 2013Generating Sequences With Recurrent Neural Networkscited by 554
14 more
  1. 2019Importance Estimation for Neural Network Pruningcited by 541
  2. 2019GhostNet: More Features from Cheap Operationscited by 410
  3. 2015Deep Learning with Limited Numerical Precisioncited by 376
  4. 2018Filter Pruning via Geometric Median for Deep Convolutional Neural Networks Accelerationcited by 367
  5. 2018Soft Filter Pruning for Accelerating Deep Convolutional Neural Networkscited by 315
  6. 2021Accelerating Sparse Deep Neural Networkscited by 223
  7. 2016Network Trimming: A Data-Driven Neuron Pruning Approach towards Efficient Deep Architecturescited by 221
  8. 2020Dynamic Model Pruning with Feedbackcited by 125
  9. 2020PatDNN: Achieving Real-Time DNN Execution on Mobile Devices with Pattern-based Weight Pruningcited by 76
  10. 2020MNN: A Universal and Efficient Inference Enginecited by 65
  11. 2021Achieving on-Mobile Real-Time Super-Resolution with Neural Architecture and Pruning Searchcited by 34
  12. 2020Convolution-Weight-Distribution Assumption: Rethinking the Criteria of Channel Pruningcited by 30
  13. 2020Comparison and Benchmarking of AI Models and Frameworks on Mobile Devicescited by 17
  14. 2019Deep Learning on Mobile Devices - A Reviewcited by 16

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2021SparseDNN: Fast Sparse Deep Learning Inference on CPUs
  2. 2021GRIM: A General, Real-Time Deep Learning Inference Framework for Mobile Devices based on Fine-Grained Structured Weight Sparsity
  3. 2026SparseDecoding: Decoding-Aware Pruning for Accurate and Efficient LLM Inference
  4. 2025BLaST: High Performance Inference and Pretraining using BLock Sparse Transformers
  5. 2026PrunePath: Towards Highly Structured Sparse Language Models
  6. 2022S4: a High-sparsity, High-performance AI Accelerator
  7. 2025Accelerating LLM Inference with Flexible N:M Sparsity via A Fully Digital Compute-in-Memory Accelerator
  8. 2026Beyond FLOPs: Benchmarking Real Inference Acceleration of LLM Pruning under a GEMM-Centric Taxonomy