mlsanityData to 14 Oct 2026

ELiTeFormer: An Efficient Transformer for FPGAs

arXiv 2607.03652 ↗First posted 4 Jul 2026cs.AR cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

Transformer blocks are prevalent in large language model (LLM) but present deployment challenges due to their challenging computational and memory demands. While prior work has typically optimized attention mechanisms or feed-forward networks (FFNs) separately, few hardware (HW) architecture have jointly addressed both components with co-designed hardware acceleration. We present ELiTeFormer (Efficient Linear Ternary Transformer), the first Transformer model architecture that unifies hybrid linear attention with ultra-low-precision (ternary) linear projections, specifically co-designed for field-programmable gate array (FPGA) deployment. ELiTeFormer achieves 10x model weight compression and 12.8x key-value (KV) cache compression compared to LLaMA 3, while maintaining competitive accuracy (31.9% on the MMLU benchmark, within 3.0% of BitNet b1.58). Our key architectural contribution is a novel processing element (PE) micro-architecture that eliminates all multiplications in ternary linear projections through bitmasking operations, significantly reducing resource utilization by completely avoiding dedicated digital signal processing (DSP) blocks. We simulate, synthesize, and deploy ELiTeFormer targeting a Xilinx VCK5000 Versal board using high-level synthesis (HLS) flows. Block-level simulations show 9.6x speedup for FFN operations and 4.4x speedup for attention compared to standard implementations. End-to-end deployment achieves up to 3.9x lower latency and 3.2x better energy efficiency than LLaMA 3 on an NVIDIA A100 graphics processing unit (GPU) at long context lengths. This represents the first FPGA realization combining linear attention with ternary quantization, demonstrating the viability of algorithm-architecture co-design for next-generation LLM acceleration.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 32 of its 57 references that are in the map. First posted on arXiv 4 Jul 2026.

Builds on

32 of 57 references.

References · 32

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2020Longformer: The Long-Document Transformercited by 3,413
  4. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 2,022
  5. 2020Big Bird: Transformers for Longer Sequencescited by 1,684
  6. 2020Rethinking Attention with Performerscited by 1,601
  7. 2020Linformer: Self-Attention with Linear Complexitycited by 1,474
  8. 2024The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scalecited by 1,160
  9. 2022LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scalecited by 906
  10. 2023Gated Linear Attention Transformers with Hardware-Efficient Trainingcited by 514
  11. 2021Nystr"omformer: A Nystr"om-Based Algorithm for Approximating Self-Attentioncited by 471
  12. 2024Parallelizing Linear Transformers with the Delta Rule over Sequence Lengthcited by 396
20 more
  1. 2024The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bitscited by 308
  2. 2023Sentiment Analysis in the Era of Large Language Models: A Reality Checkcited by 268
  3. 2020Sparse Sinkhorn Attentioncited by 251
  4. 2020SpAtten: Efficient Sparse Attention Architecture with Cascade Token and Head Pruningcited by 234
  5. 2024Simple linear attention language models balance the recall-throughput tradeoffcited by 208
  6. 2024The Hedgehog & the Porcupine: Expressive Linear Attentions with Softmax Mimicrycited by 120
  7. 2021Long-Short Transformer: Efficient Transformers for Language and Visioncited by 102
  8. 2024Get More with LESS: Synthesizing Recurrence with KV Cache Compression for Efficient LLM Inferencecited by 75
  9. 2024FlightLLM: Efficient Large Language Model Inference with a Complete Mapping Flow on FPGAscited by 64
  10. 2020FTRANS: Energy-Efficient Acceleration of Transformers using FPGAcited by 54
  11. 2023PolySketchFormer: Fast Transformers via Sketching Polynomial Kernelscited by 36
  12. 2022Adaptable Butterfly Accelerator for Attention-based NNs via Hardware and Algorithm Co-designcited by 31
  13. 2022An Algorithm-Hardware Co-Optimized Framework for Accelerating N:M Sparse Transformerscited by 29
  14. 2022ViTALiTy: Unifying Low-rank and Sparse Approximation for Vision Transformer Acceleration with a Linear Taylor Attentioncited by 29
  15. 2023Edge-MoE: Memory-Efficient Multi-Task Vision Transformer Architecture with Task-level Sparsity via Mixture-of-Expertscited by 23
  16. 2021NPE: An FPGA-based Overlay Processor for Natural Language Processingcited by 17
  17. 2022Dynamic N:M Fine-grained Structured Sparse Attention Mechanismcited by 14
  18. 2022SALO: An Efficient Spatial Accelerator Enabling Hybrid Sparse Attention Mechanisms for Long Sequencescited by 10
  19. 2023Improving Autoregressive NLP Tasks via Modular Linearized Attentioncited by 2
  20. 2024LeaPformer: Enabling Linear Transformers for Autoregressive and Simultaneous Tasks via Learned Proportionscited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025TeLLMe: An Energy-Efficient Ternary LLM Accelerator for Prefilling and Decoding on Edge FPGAs
  2. 2026Lightweight Transformer Architectures for Edge Devices in Real-Time Applications
  3. 2025On-Device Qwen2.5: Efficient LLM Inference with Model Compression and Hardware Acceleration
  4. 2026Transformer Accelerator (TFA): A Macro-Op INT8 Hardware Chip for Transformer Inference and Machine Translation
  5. 2024TATAA: Programmable Mixed-Precision Transformer Acceleration with a Transformable Arithmetic Architecture
  6. 2024Resource-Efficient Transformer Architecture: Optimizing Memory and Execution Time for Real-Time Applications
  7. 2025Optimization Strategies for Enhancing Resource Efficiency in Transformers & Large Language Models
  8. 2024A Runtime-Adaptive Transformer Neural Network Accelerator on FPGAs