mlsanityData to 7 Oct 2026

Benchmarking TinyML Systems: Challenges and Direction

arXiv 2003.04821 ↗First posted 10 Mar 2020cs.PF cs.LGcited by 89

Not on the map. Why?

Abstract (arXiv)

Recent advancements in ultra-low-power machine learning (TinyML) hardware promises to unlock an entirely new class of smart applications. However, continued progress is limited by the lack of a widely accepted benchmark for these systems. Benchmarking allows us to measure and thereby systematically compare, evaluate, and improve the performance of systems and is therefore fundamental to a field reaching maturity. In this position paper, we present the current landscape of TinyML and discuss the challenges and direction towards developing a fair and useful hardware benchmark for TinyML workloads. Furthermore, we present our four benchmarks and discuss our selection methodology. Our viewpoints reflect the collective thoughts of the TinyMLPerf working group that is comprised of over 30 organizations.

Key facts. Cited by 89 papers in the map (data through 7 Oct 2026). First posted on arXiv 10 Mar 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 19; 2022: 21; 2023: 11; 2024: 13; 2025: 14; 2026: 11.

89 papers.

Sort:

By area:

  1. 2022On-Device Training Under 256KB Memorycited by 158
  2. 2021A Survey on Hyperdimensional Computing aka Vector Symbolic Architectures, Part II: Applications, Cognitive Models, and Challengescited by 68
  3. 2021hls4ml: An Open-Source Codesign Workflow to Empower Scientific Low-Power Machine Learning Devicescited by 61
  4. 2021A TinyML Platform for On-Device Continual Learning with Quantized Latent Replayscited by 37
  5. 2021DeepEdgeBench: Benchmarking Deep Neural Networks on Edge Devicescited by 26
  6. 2023A Machine Learning-oriented Survey on Tiny Machine Learningcited by 26
  7. 2023PockEngine: Sparse and Efficient Fine-tuning in a Pocketcited by 22
  8. 2021Bottom-up and top-down approaches for the design of neuromorphic processing systems: Tradeoffs and synergies between natural and artificial intelligencecited by 21
  9. 2022A Survey of Graph-based Deep Learning for Anomaly Detection in Distributed Systemscited by 20
  10. 2021Widening Access to Applied Machine Learning with TinyMLcited by 17
  11. 2022Tiny Robot Learning: Challenges and Directions for Machine Learning in Resource-Constrained Robotscited by 17
  12. 2022P2M: A Processing-in-Pixel-in-Memory Paradigm for Resource-Constrained TinyML Applicationscited by 16
38 more
  1. 2023Lightweight Neural Architecture Search for Temporal Convolutional Networks at the Edgecited by 15
  2. 2023TinyML: Tools, Applications, Challenges, and Future Research Directionscited by 14
  3. 2021TinyML for Ubiquitous Edge AIcited by 13
  4. 2022Intelligence at the Extreme Edge: A Survey on Reformable TinyMLcited by 13
  5. 2021Generalized Learning Vector Quantization for Classification in Randomized Neural Networks and Hyperdimensional Computingcited by 12
  6. 2022CFU Playground: Full-Stack Open-Source Framework for Tiny Machine Learning (tinyML) Acceleration on FPGAscited by 11
  7. 2022Real-time semantic segmentation on FPGAs for autonomous vehicles with hls4mlcited by 11
  8. 2023Reduced Precision Floating-Point Optimization for Deep Neural Network On-Device Learning on MicroControllerscited by 10
  9. 2022Open-source FPGA-ML codesign for the MLPerf Tiny Benchmarkcited by 9
  10. 2024On TinyML and Cybersecurity: Electric Vehicle Charging Infrastructure Use Casecited by 9
  11. 2021Measuring what Really Matters: Optimizing Neural Networks for TinyMLcited by 8
  12. 2022Tiny, always-on and fragile: Bias propagation through design choices in on-device machine learning workflowscited by 8
  13. 2024Optimizing the Deployment of Tiny Transformers on Low-Power MCUscited by 8
  14. 2021Tiny Machine Learning for Concept Driftcited by 7
  15. 2021Revisiting IoT Device Identificationcited by 6
  16. 2021TinyFedTL: Federated Transfer Learning on Tiny Devicescited by 6
  17. 2024Talaria: Interactively Optimizing Machine Learning Models for Efficient Inferencecited by 6
  18. 2021TinyML Platforms Benchmarkingcited by 5
  19. 2022Machine Learning Sensorscited by 5
  20. 2023U-TOE: Universal TinyML On-board Evaluation Toolkit for Low-Power IoTcited by 5
  21. 2025From Tiny Machine Learning to Tiny Deep Learning: A Surveycited by 5
  22. 2021TENT: Efficient Quantization of Neural Networks on the tiny Edge with Tapered FixEd PoiNTcited by 4
  23. 2022Depth Pruning with Auxiliary Networks for TinyMLcited by 4
  24. 2023Reaching the Edge of the Edge: Image Analysis in Spacecited by 4
  25. 2023Sharing Low Rank Conformer Weights for Tiny Always-On Ambient Speech Recognition Modelscited by 4
  26. 2024DTMM: Deploying TinyML Models on Extremely Weak IoT Devices with Pruningcited by 4
  27. 2024Enabling On-device Continual Learning with Binary Neural Networkscited by 4
  28. 2024Boosting keyword spotting through on-device learnable user speech characteristicscited by 4
  29. 2024TinyChirp: Bird Song Recognition Using TinyML Models on Low-power Wireless Acoustic Sensorscited by 4
  30. 2025ETHEREAL: Energy-efficient and High-throughput Inference using Compressed Tsetlin Machinecited by 4
  31. 2025Benchmarking Energy and Latency in TinyML: A Novel Method for Resource-Constrained AIcited by 4
  32. 2022DeepPicarMicro: Applying TinyML to Autonomous Cyber Physical Systemscited by 3
  33. 202412 mJ per Class On-Device Online Few-Shot Class-Incremental Learningcited by 3
  34. 2021TiWS-iForest: Isolation Forest in Weakly Supervised and Tiny ML scenarioscited by 2
  35. 2021Energy-Efficient Inference on the Edge Exploiting TinyML Capabilities for UAVscited by 2
  36. 2022Software Engineering Approaches for TinyML based IoT Embedded Vision: A Systematic Literature Reviewcited by 2
  37. 2022Smart City Intersections: Intelligence Nodes for Future Metropolisescited by 2
  38. 2022Multi-Complexity-Loss DNAS for Energy-Efficient and Memory-Constrained Deep Neural Networkscited by 2

50 most cited of 89 All 89 (JSON)