# TinyTorch: Building Machine Learning Systems from First Principles

arXiv: https://arxiv.org/abs/2601.19107 · HTML: https://mlsanity.com/paper/2601.19107/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 10 of its 46 references that are in the map.
- First posted on arXiv 27 Jan 2026.

## Builds on (most cited first, 10 of 10)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Energy and Policy Considerations for Deep Learning in NLP](https://mlsanity.com/paper/1906.02243/) (2019)
- [Carbon Emissions and Large Neural Network Training](https://mlsanity.com/paper/2104.10350/) (2021)
- [MLPerf Inference Benchmark](https://mlsanity.com/paper/1911.02549/) (2019)
- [fastai: A Layered API for Deep Learning](https://mlsanity.com/paper/2002.04688/) (2020)
- [MLPerf Training Benchmark](https://mlsanity.com/paper/1910.01500/) (2019)
- [Dive into Deep Learning](https://mlsanity.com/paper/2106.11342/) (2021)
- [Benchmarking TinyML Systems: Challenges and Direction](https://mlsanity.com/paper/2003.04821/) (2020)
- [ASTRA-sim2.0: Modeling Hierarchical Networks and Disaggregated Systems for Large-model Training at Scale](https://mlsanity.com/paper/2303.14006/) (2023)
- [Widening Access to Applied Machine Learning with TinyML](https://mlsanity.com/paper/2106.04008/) (2021)

## Built on by (most cited first, 1 of 1)

- [On-Device Vision Training, Deployment, and Inference on a Thumb-Sized Microcontroller](https://mlsanity.com/paper/2604.23012/) (2026)

## Abstract (arXiv)

Machine learning education faces a fundamental gap: students learn algorithms without understanding the systems that execute them. They study gradient descent without measuring memory, attention mechanisms without analyzing O(N^2) scaling, optimizer theory without knowing why Adam requires 3x the memory of SGD. This "algorithm-systems divide" produces practitioners who can train models but cannot debug memory failures, optimize inference latency, or reason about deployment trade-offs--the very skills industry demands as "ML systems engineering." We present TinyTorch, a 20-module curriculum that closes this gap through "implementation-based systems pedagogy": students construct PyTorch's core components (tensors, autograd, optimizers, CNNs, transformers) in pure Python, building a complete framework where every operation they invoke is code they wrote. The design employs three patterns: "progressive disclosure" of complexity, "systems-first integration" of profiling from the first module, and "build-to-validate milestones" recreating 67 years of ML breakthroughs--from Perceptron (1958) through Transformers (2017) to MLPerf-style benchmarking. Requiring only 4GB RAM and no GPU, TinyTorch demonstrates that deep ML systems understanding is achievable without specialized hardware. The curriculum is available open-source at mlsysbook.ai/tinytorch.