# Benchmarking TinyML Systems: Challenges and Direction

arXiv: https://arxiv.org/abs/2003.04821 · HTML: https://mlsanity.com/paper/2003.04821/

## Key facts

- Cited by 89 papers in the map (data through 7 Oct 2026).
- First posted on arXiv 10 Mar 2020.

## Built on by (most cited first, 30 of 89)

- [On-Device Training Under 256KB Memory](https://mlsanity.com/paper/2206.15472/) (2022)
- [A Survey on Hyperdimensional Computing aka Vector Symbolic Architectures, Part II: Applications, Cognitive Models, and Challenges](https://mlsanity.com/paper/2112.15424/) (2021)
- [hls4ml: An Open-Source Codesign Workflow to Empower Scientific Low-Power Machine Learning Devices](https://mlsanity.com/paper/2103.05579/) (2021)
- [A TinyML Platform for On-Device Continual Learning with Quantized Latent Replays](https://mlsanity.com/paper/2110.10486/) (2021)
- [DeepEdgeBench: Benchmarking Deep Neural Networks on Edge Devices](https://mlsanity.com/paper/2108.09457/) (2021)
- [A Machine Learning-oriented Survey on Tiny Machine Learning](https://mlsanity.com/paper/2309.11932/) (2023)
- [PockEngine: Sparse and Efficient Fine-tuning in a Pocket](https://mlsanity.com/paper/2310.17752/) (2023)
- [Bottom-up and top-down approaches for the design of neuromorphic processing systems: Tradeoffs and synergies between natural and artificial intelligence](https://mlsanity.com/paper/2106.01288/) (2021)
- [A Survey of Graph-based Deep Learning for Anomaly Detection in Distributed Systems](https://mlsanity.com/paper/2206.04149/) (2022)
- [Widening Access to Applied Machine Learning with TinyML](https://mlsanity.com/paper/2106.04008/) (2021)
- [Tiny Robot Learning: Challenges and Directions for Machine Learning in Resource-Constrained Robots](https://mlsanity.com/paper/2205.05748/) (2022)
- [P2M: A Processing-in-Pixel-in-Memory Paradigm for Resource-Constrained TinyML Applications](https://mlsanity.com/paper/2203.04737/) (2022)
- [Lightweight Neural Architecture Search for Temporal Convolutional Networks at the Edge](https://mlsanity.com/paper/2301.10281/) (2023)
- [TinyML: Tools, Applications, Challenges, and Future Research Directions](https://mlsanity.com/paper/2303.13569/) (2023)
- [TinyML for Ubiquitous Edge AI](https://mlsanity.com/paper/2102.01255/) (2021)
- [Intelligence at the Extreme Edge: A Survey on Reformable TinyML](https://mlsanity.com/paper/2204.00827/) (2022)
- [Generalized Learning Vector Quantization for Classification in Randomized Neural Networks and Hyperdimensional Computing](https://mlsanity.com/paper/2106.09821/) (2021)
- [CFU Playground: Full-Stack Open-Source Framework for Tiny Machine Learning (tinyML) Acceleration on FPGAs](https://mlsanity.com/paper/2201.01863/) (2022)
- [Real-time semantic segmentation on FPGAs for autonomous vehicles with hls4ml](https://mlsanity.com/paper/2205.07690/) (2022)
- [Reduced Precision Floating-Point Optimization for Deep Neural Network On-Device Learning on MicroControllers](https://mlsanity.com/paper/2305.19167/) (2023)
- [Open-source FPGA-ML codesign for the MLPerf Tiny Benchmark](https://mlsanity.com/paper/2206.11791/) (2022)
- [On TinyML and Cybersecurity: Electric Vehicle Charging Infrastructure Use Case](https://mlsanity.com/paper/2404.16894/) (2024)
- [Measuring what Really Matters: Optimizing Neural Networks for TinyML](https://mlsanity.com/paper/2104.10645/) (2021)
- [Tiny, always-on and fragile: Bias propagation through design choices in on-device machine learning workflows](https://mlsanity.com/paper/2201.07677/) (2022)
- [Optimizing the Deployment of Tiny Transformers on Low-Power MCUs](https://mlsanity.com/paper/2404.02945/) (2024)
- [Tiny Machine Learning for Concept Drift](https://mlsanity.com/paper/2107.14759/) (2021)
- [Revisiting IoT Device Identification](https://mlsanity.com/paper/2107.07818/) (2021)
- [TinyFedTL: Federated Transfer Learning on Tiny Devices](https://mlsanity.com/paper/2110.01107/) (2021)
- [Talaria: Interactively Optimizing Machine Learning Models for Efficient Inference](https://mlsanity.com/paper/2404.03085/) (2024)
- [TinyML Platforms Benchmarking](https://mlsanity.com/paper/2112.01319/) (2021)

## Abstract (arXiv)

Recent advancements in ultra-low-power machine learning (TinyML) hardware promises to unlock an entirely new class of smart applications. However, continued progress is limited by the lack of a widely accepted benchmark for these systems. Benchmarking allows us to measure and thereby systematically compare, evaluate, and improve the performance of systems and is therefore fundamental to a field reaching maturity. In this position paper, we present the current landscape of TinyML and discuss the challenges and direction towards developing a fair and useful hardware benchmark for TinyML workloads. Furthermore, we present our four benchmarks and discuss our selection methodology. Our viewpoints reflect the collective thoughts of the TinyMLPerf working group that is comprised of over 30 organizations.