mlsanityData to 14 Oct 2026

Vis-TOP: Visual Transformer Overlay Processor

arXiv 2110.10957 ↗First posted 21 Oct 2021cs.CV cs.ARcited by 2

Not on the map. Why?

Abstract (arXiv)

In recent years, Transformer has achieved good results in Natural Language Processing (NLP) and has also started to expand into Computer Vision (CV). Excellent models such as the Vision Transformer and Swin Transformer have emerged. At the same time, the platform for Transformer models was extended to embedded devices to meet some resource-sensitive application scenarios. However, due to the large number of parameters, the complex computational flow and the many different structural variants of Transformer models, there are a number of issues that need to be addressed in its hardware design. This is both an opportunity and a challenge. We propose Vis-TOP (Visual Transformer Overlay Processor), an overlay processor for various visual Transformer models. It differs from coarse-grained overlay processors such as CPU, GPU, NPE, and from fine-grained customized designs for a specific model. Vis-TOP summarizes the characteristics of all visual Transformer models and implements a three-layer and two-level transformation structure that allows the model to be switched or changed freely without changing the hardware architecture. At the same time, the corresponding instruction bundle and hardware architecture are designed in three-layer and two-level transformation structure. After quantization of Swin Transformer tiny model using 8-bit fixed points (fix_8), we implemented an overlay processor on the ZCU102. Compared to GPU, the TOP throughput is 1.5x higher. Compared to the existing Transformer accelerators, our throughput per DSP is between 2.2x and 11.7x higher than others. In a word, the approach in this paper meets the requirements of real-time AI in terms of both resource consumption and inference speed. Vis-TOP provides a cost-effective and power-effective solution based on reconfigurable devices for computer vision at the edge.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 17 of its 28 references that are in the map. First posted on arXiv 21 Oct 2021.

Builds on

17 of 28 references.

References · 17

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  4. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  5. 2020Training data-efficient image transformers & distillation through attentioncited by 4,586
  6. 2016Gaussian Error Linear Units (GELUs)cited by 3,502
  7. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,790
  8. 2019TinyBERT: Distilling BERT for Natural Language Understandingcited by 1,167
  9. 2021ConViT: Improving Vision Transformers with Soft Convolutional Inductive Biasescited by 484
  10. 2020MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devicescited by 482
  11. 2020Lite Transformer with Long-Short Range Attentioncited by 127
  12. 2020FTRANS: Energy-Efficient Acceleration of Transformers using FPGAcited by 54
5 more
  1. 2020Hardware Accelerator for Multi-Head Attention and Position-Wise Feed-Forward in the Transformercited by 43
  2. 2021Hardware Acceleration of Fully Quantized BERT for Efficient Natural Language Processingcited by 22
  3. 2018Synetgy: Algorithm-hardware Co-design for ConvNet Accelerators on Embedded FPGAscited by 18
  4. 2021NPE: An FPGA-based Overlay Processor for Natural Language Processingcited by 17
  5. 2018DLA: Compiler and FPGA Overlay for Neural Network Inference Accelerationcited by 15

Built on by

Cited by year

Citing papers per year: 2022: 1; 2023: 0; 2024: 1.

2 papers.

  1. 2022Row-wise Accelerator for Vision Transformercited by 2
  2. 2024From Algorithm to Hardware: A Survey on Efficient and Safe Deployment of Deep Neural Networkscited by 2

Similar, not cited

Close in wording, no citation either way

  1. 2023A survey on efficient vision transformers: algorithms, techniques, and performance benchmarking
  2. 2024SOLO: A Single Transformer for Scalable Vision-Language Modeling
  3. 2022VAQF: Fully Automatic Software-Hardware Co-Design Framework for Low-Bit Vision Transformer
  4. 2023Edge-MoE: Memory-Efficient Multi-Task Vision Transformer Architecture with Task-level Sparsity via Mixture-of-Experts
  5. 2024ViTamin: Designing Scalable Vision Models in the Vision-Language Era
  6. 2024Model Quantization and Hardware Acceleration for Vision Transformers: A Comprehensive Survey
  7. 2024Trio-ViT: Post-Training Quantization and Acceleration for Softmax-Free Efficient Vision Transformer
  8. 2025Real Time FPGA Based Transformers & VLMs for Vision Tasks: SOTA Designs and Optimizations