# CHOSEN: Compilation to Hardware Optimization Stack for Efficient Vision Transformer Inference

arXiv: https://arxiv.org/abs/2407.12736 · HTML: https://mlsanity.com/paper/2407.12736/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 17 references that are in the map.
- First posted on arXiv 17 Jul 2024.

## Builds on (most cited first, 14 of 14)

- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Training data-efficient image transformers & distillation through attention](https://mlsanity.com/paper/2012.12877/) (2020)
- [Post-Training Quantization for Vision Transformer](https://mlsanity.com/paper/2106.14156/) (2021)
- [ViTCoD: Vision Transformer Acceleration via Dedicated Algorithm and Accelerator Co-Design](https://mlsanity.com/paper/2210.09573/) (2022)
- [Auto-ViT-Acc: An FPGA-Aware Automatic Acceleration Framework for Vision Transformer with Mixed-Scheme Quantization](https://mlsanity.com/paper/2208.05163/) (2022)
- [VAQF: Fully Automatic Software-Hardware Co-Design Framework for Low-Bit Vision Transformer](https://mlsanity.com/paper/2201.06618/) (2022)
- [PEANO-ViT: Power-Efficient Approximations of Non-Linearities in Vision Transformers](https://mlsanity.com/paper/2406.14854/) (2024)
- [ME-ViT: A Single-Load Memory-Efficient FPGA Accelerator for Vision Transformers](https://mlsanity.com/paper/2402.09709/) (2024)
- [NeuroBlend: Towards Low-Power yet Accurate Neural Network-Based Inference Engine Blending Binary and Fixed-Point Convolutions](https://mlsanity.com/paper/2307.03784/) (2023)
- [Memory-Efficient Vision Transformers: An Activation-Aware Mixed-Rank Compression Strategy](https://mlsanity.com/paper/2402.06004/) (2024)
- [Training-Free Acceleration of ViTs with Delayed Spatial Merging](https://mlsanity.com/paper/2303.02331/) (2023)
- [Flexible Communication Avoiding Matrix Multiplication on FPGA with High-Level Synthesis](https://mlsanity.com/paper/1912.06526/) (2019)
- [CADC: Encoding User-Item Interactions for Compressing Recommendation Model Training Data](https://mlsanity.com/paper/2407.08108/) (2024)

## Built on by (most cited first, 2 of 2)

- [Fever Detection with Infrared Thermography: Enhancing Accuracy through Machine Learning Techniques](https://mlsanity.com/paper/2407.15302/) (2024)
- [Efficient Noise Mitigation for Enhancing Inference Accuracy in DNNs on Mixed-Signal Accelerators](https://mlsanity.com/paper/2409.18553/) (2024)

## Abstract (arXiv)

Vision Transformers (ViTs) represent a groundbreaking shift in machine learning approaches to computer vision. Unlike traditional approaches, ViTs employ the self-attention mechanism, which has been widely used in natural language processing, to analyze image patches. Despite their advantages in modeling visual tasks, deploying ViTs on hardware platforms, notably Field-Programmable Gate Arrays (FPGAs), introduces considerable challenges. These challenges stem primarily from the non-linear calculations and high computational and memory demands of ViTs. This paper introduces CHOSEN, a software-hardware co-design framework to address these challenges and offer an automated framework for ViT deployment on the FPGAs in order to maximize performance. Our framework is built upon three fundamental contributions: multi-kernel design to maximize the bandwidth, mainly targeting benefits of multi DDR memory banks, approximate non-linear functions that exhibit minimal accuracy degradation, and efficient use of available logic blocks on the FPGA, and efficient compiler to maximize the performance and memory-efficiency of the computing kernels by presenting a novel algorithm for design space exploration to find optimal hardware configuration that achieves optimal throughput and latency. Compared to the state-of-the-art ViT accelerators, CHOSEN achieves a 1.5x and 1.42x improvement in the throughput on the DeiT-S and DeiT-B models.