# Resource-aware Mixed-precision Quantization for Enhancing Deployability of Transformers for Time-series Forecasting on Embedded FPGAs

arXiv: https://arxiv.org/abs/2410.03294 · HTML: https://mlsanity.com/paper/2410.03294/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 21 references that are in the map.
- First posted on arXiv 4 Oct 2024.

## Builds on (most cited first, 15 of 15)

- [Transformers in Time Series: A Survey](https://mlsanity.com/paper/2202.07125/) (2022)
- [A White Paper on Neural Network Quantization](https://mlsanity.com/paper/2106.08295/) (2021)
- [HAWQ-V2: Hessian Aware trace-Weighted Quantization of Neural Networks](https://mlsanity.com/paper/1911.03852/) (2019)
- [Edge Intelligence: Paving the Last Mile of Artificial Intelligence with Edge Computing](https://mlsanity.com/paper/1905.10083/) (2019)
- [Large Language Models for Time Series: A Survey](https://mlsanity.com/paper/2402.01801/) (2024)
- [A Survey of Techniques for Optimizing Transformer Inference](https://mlsanity.com/paper/2307.07982/) (2023)
- [Auto-ViT-Acc: An FPGA-Aware Automatic Acceleration Framework for Vision Transformer with Mixed-Scheme Quantization](https://mlsanity.com/paper/2208.05163/) (2022)
- [4-bit Conformer with Native Quantization Aware Training for Speech Recognition](https://mlsanity.com/paper/2203.15952/) (2022)
- [AI for Next Generation Computing: Emerging Trends and Future Directions](https://mlsanity.com/paper/2203.04159/) (2022)
- [RMSMP: A Novel Deep Neural Network Quantization Framework with Row-wise Mixed Schemes and Multiple Precisions](https://mlsanity.com/paper/2111.00153/) (2021)
- [Enhancing Energy-efficiency by Solving the Throughput Bottleneck of LSTM Cells for Embedded FPGAs](https://mlsanity.com/paper/2310.16842/) (2023)
- [Sparse-VQ Transformer: An FFN-Free Framework with Vector Quantization for Enhanced Time Series Forecasting](https://mlsanity.com/paper/2402.05830/) (2024)
- [Integer-only Quantized Transformers for Embedded FPGA-based Time-series Forecasting in AIoT](https://mlsanity.com/paper/2407.11041/) (2024)
- [ElasticAI: Creating and Deploying Energy-Efficient Deep Learning Accelerator for Pervasive Computing](https://mlsanity.com/paper/2409.09044/) (2024)
- [The Benefits of Edge Computing in Healthcare, Smart Cities, and IoT](https://mlsanity.com/paper/2112.01250/) (2021)

## Built on by (most cited first, 1 of 1)

- [Automating Versatile Time-Series Analysis with Tiny Transformers on Embedded FPGAs](https://mlsanity.com/paper/2505.17662/) (2025)

## Abstract (arXiv)

This study addresses the deployment challenges of integer-only quantized Transformers on resource-constrained embedded FPGAs (Xilinx Spartan-7 XC7S15). We enhanced the flexibility of our VHDL template by introducing a selectable resource type for storing intermediate results across model layers, thereby breaking the deployment bottleneck by utilizing BRAM efficiently. Moreover, we developed a resource-aware mixed-precision quantization approach that enables researchers to explore hardware-level quantization strategies without requiring extensive expertise in Neural Architecture Search. This method provides accurate resource utilization estimates with a precision discrepancy as low as 3%, compared to actual deployment metrics. Compared to previous work, our approach has successfully facilitated the deployment of model configurations utilizing mixed-precision quantization, thus overcoming the limitations inherent in five previously non-deployable configurations with uniform quantization bitwidths. Consequently, this research enhances the applicability of Transformers in embedded systems, facilitating a broader range of Transformer-powered applications on edge devices.