mlsanityData to 7 Oct 2026

HG-PIPE: Vision Transformer Acceleration with Hybrid-Grained Pipeline

arXiv 2407.17879 ↗First posted 25 Jul 2024cs.AR cs.AIcited by 7

Not on the map. Why?

Abstract (arXiv)

Vision Transformer (ViT) acceleration with field programmable gate array (FPGA) is promising but challenging. Existing FPGA-based ViT accelerators mainly rely on temporal architectures, which process different operators by reusing the same hardware blocks and suffer from extensive memory access overhead. Pipelined architectures, either coarse-grained or fine-grained, unroll the ViT computation spatially for memory access efficiency. However, they usually suffer from significant hardware resource constraints and pipeline bubbles induced by the global computation dependency of ViT. In this paper, we introduce HG-PIPE, a pipelined FPGA accelerator for high-throughput and low-latency ViT processing. HG-PIPE features a hybrid-grained pipeline architecture to reduce on-chip buffer cost and couples the computation dataflow and parallelism design to eliminate the pipeline bubbles. HG-PIPE further introduces careful approximations to implement both linear and non-linear operators with abundant Lookup Tables (LUTs), thus alleviating resource constraints. On a ZCU102 FPGA, HG-PIPE achieves 2.78 times better throughput and 2.52 times better resource efficiency than the prior-art accelerators, e.g., AutoViTAcc. With a VCK190 FPGA, HG-PIPE realizes end-to-end ViT acceleration on a single device and achieves 7118 images/s, which is 2.81 times faster than a V100 GPU.

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Builds on 20 of its 51 references that are in the map. First posted on arXiv 25 Jul 2024.

Builds on

20 of 51 references.

References · 20

  1. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  2. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  3. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  4. 2020End-to-End Object Detection with Transformerscited by 7,384
  5. 2015You Only Look Once: Unified, Real-Time Object Detectioncited by 4,829
  6. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  7. 2021Segmenter: Transformer for Semantic Segmentationcited by 855
  8. 2021I-BERT: Integer-only BERT Quantizationcited by 250
  9. 2022HeatViT: Hardware-Efficient Adaptive Token Pruning for Vision Transformerscited by 46
  10. 2022Adaptable Butterfly Accelerator for Attention-based NNs via Hardware and Algorithm Co-designcited by 29
  11. 2023Understanding the Potential of FPGA-Based Spatial Acceleration for Large Language Model Inferencecited by 29
  12. 2022Auto-ViT-Acc: An FPGA-Aware Automatic Acceleration Framework for Vision Transformer with Mixed-Scheme Quantizationcited by 28
8 more
  1. 2022VAQF: Fully Automatic Software-Hardware Co-Design Framework for Low-Bit Vision Transformercited by 27
  2. 2023AccelTran: A Sparsity-Aware Accelerator for Dynamic Inference with Transformerscited by 24
  3. 2019LUTNet: Learning FPGA Configurations for Highly Efficient Neural Network Inferencecited by 23
  4. 2023Edge-MoE: Memory-Efficient Multi-Task Vision Transformer Architecture with Task-level Sparsity via Mixture-of-Expertscited by 23
  5. 2021Hardware Acceleration of Fully Quantized BERT for Efficient Natural Language Processingcited by 21
  6. 2022A Length Adaptive Algorithm-Hardware Co-design of Transformer on FPGA Through Sparse Attention and Dynamic Pipeliningcited by 21
  7. 2022DeFiNES: Enabling Fast Exploration of the Depth-first Scheduling Space for DNN Accelerators through Analytical Modelingcited by 12
  8. 2024SSR: Spatial Sequential Hybrid Architecture for Latency Throughput Tradeoff in Transformer Accelerationcited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 6; 2026: 1.

7 papers.

  1. 2025LightMamba: Efficient Mamba Acceleration on FPGA with Quantization and Hardware Co-designcited by 11
  2. 2025LL-ViT: Edge Deployable Vision Transformers with Look Up Table Neuronscited by 3
  3. 2025FPQVAR: Floating Point Quantization for Visual Autoregressive Model with FPGA Hardware Co-designcited by 0
  4. 2025CoQMoE: Co-Designed Quantization and Computation Orchestration for Mixture-of-Experts Vision Transformer on FPGAcited by 0
  5. 2025TrackCore-F: Deploying Transformer-Based Subatomic Particle Tracking on FPGAscited by 0
  6. 2025QUILL: An Algorithm-Architecture Co-Design for Cache-Local Deformable Attentioncited by 0
  7. 2026DAPA: Distribution Aware Piecewise Activation Functions for On-Device Transformer Inference and Trainingcited by 0