# A Hardware-Software Blueprint for Flexible Deep Learning Specialization

arXiv: https://arxiv.org/abs/1807.04188 · HTML: https://mlsanity.com/paper/1807.04188/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 11 Jul 2018.

## Built on by (most cited first, 7 of 7)

- [TensorIR: An Abstraction for Automatic Tensorized Program Optimization](https://mlsanity.com/paper/2207.04296/) (2022)
- [Bring Your Own Codegen to Deep Learning Compiler](https://mlsanity.com/paper/2105.03215/) (2021)
- [A Highly Configurable Hardware/Software Stack for DNN Inference Acceleration](https://mlsanity.com/paper/2111.15024/) (2021)
- [Bifrost: End-to-End Evaluation and Optimization of Reconfigurable DNN Accelerators](https://mlsanity.com/paper/2204.12418/) (2022)
- [Tackling the Matrix Multiplication Micro-kernel Generation with Exo](https://mlsanity.com/paper/2310.17408/) (2023)
- [Efficient Edge AI: Deploying Convolutional Neural Networks on FPGA with the Gemmini Accelerator](https://mlsanity.com/paper/2408.07404/) (2024)
- [Obsidian: Cooperative State-Space Exploration for Performant Inference on Secure ML Accelerators](https://mlsanity.com/paper/2409.02817/) (2024)

## Abstract (arXiv)

Specialized Deep Learning (DL) acceleration stacks, designed for a specific set of frameworks, model architectures, operators, and data types, offer the allure of high performance while sacrificing flexibility. Changes in algorithms, models, operators, or numerical systems threaten the viability of specialized hardware accelerators. We propose VTA, a programmable deep learning architecture template designed to be extensible in the face of evolving workloads. VTA achieves this flexibility via a parametrizable architecture, two-level ISA, and a JIT compiler. The two-level ISA is based on (1) a task-ISA that explicitly orchestrates concurrent compute and memory tasks and (2) a microcode-ISA which implements a wide variety of operators with single-cycle tensor-tensor operations. Next, we propose a runtime system equipped with a JIT compiler for flexible code-generation and heterogeneous execution that enables effective use of the VTA architecture. VTA is integrated and open-sourced into Apache TVM, a state-of-the-art deep learning compilation stack that provides flexibility for diverse models and divergent hardware backends. We propose a flow that performs design space exploration to generate a customized hardware architecture and software operator library that can be leveraged by mainstream learning frameworks. We demonstrate our approach by deploying optimized deep learning models used for object classification and style transfer on edge-class FPGAs.