mlsanityData to 14 Oct 2026

TReX- Reusing Vision Transformer's Attention for Efficient Xbar-based Computing

arXiv 2408.12742 ↗First posted 22 Aug 2024cs.AI cs.ARcited by 2

Not on the map. Why?

Abstract (arXiv)

Due to the high computation overhead of Vision Transformers (ViTs), In-memory Computing architectures are being researched towards energy-efficient deployment in edge-computing scenarios. Prior works have proposed efficient algorithm-hardware co-design and IMC-architectural improvements to improve the energy-efficiency of IMC-implemented ViTs. However, all prior works have neglected the overhead and co-depencence of attention blocks on the accuracy-energy-delay-area of IMC-implemented ViTs. To this end, we propose TReX- an attention-reuse-driven ViT optimization framework that effectively performs attention reuse in ViT models to achieve optimal accuracy-energy-delay-area tradeoffs. TReX optimally chooses the transformer encoders for attention reuse to achieve near iso-accuracy performance while meeting the user-specified delay requirement. Based on our analysis on the Imagenet-1k dataset, we find that TReX achieves 2.3x (2.19x) EDAP reduction and 1.86x (1.79x) TOPS/mm2 improvement with ~1% accuracy drop in case of DeiT-S (LV-ViT-S) ViT models. Additionally, TReX achieves high accuracy at high EDAP reduction compared to state-of-the-art token pruning and weight sharing approaches. On NLP tasks such as CoLA, TReX leads to 2% higher non-ideal accuracy compared to baseline at 1.6x lower EDAP.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 29 of its 48 references that are in the map. First posted on arXiv 22 Aug 2024.

Builds on

29 of 48 references.

References · 29

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2020Training data-efficient image transformers & distillation through attentioncited by 4,586
  3. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,611
  4. 2021Vision Transformers for Dense Predictioncited by 1,803
  5. 2021Scaling Vision Transformerscited by 999
  6. 2018Neural Network Acceptability Judgmentscited by 958
  7. 2021DynamicViT: Efficient Vision Transformers with Dynamic Token Sparsificationcited by 864
  8. 2020SpAtten: Efficient Sparse Attention Architecture with Cascade Token and Head Pruningcited by 234
  9. 2021AdaViT: Adaptive Vision Transformers for Efficient Image Recognitioncited by 228
  10. 2021Evo-ViT: Slow-Fast Token Evolution for Dynamic Vision Transformercited by 177
  11. 2021All Tokens Matter: Token Labeling for Training Better Vision Transformerscited by 144
  12. 2022EdgeNeXt: Efficiently Amalgamated CNN-Transformer Architecture for Mobile Vision Applicationscited by 121
17 more
  1. 2022MiniViT: Compressing Vision Transformers with Weight Multiplexingcited by 88
  2. 2020FTRANS: Energy-Efficient Acceleration of Transformers using FPGAcited by 54
  3. 2021Softermax: Hardware/Software Co-Design of an Efficient Softmax for Transformerscited by 54
  4. 2022HeatViT: Hardware-Efficient Adaptive Token Pruning for Vision Transformerscited by 46
  5. 2023Hardware-aware training for large-scale and diverse deep learning inference workloads using in-memory computing-based acceleratorscited by 43
  6. 2023Xformer: Hybrid X-Shaped Transformer for Image Denoisingcited by 37
  7. 2020DNN+NeuroSim V2.0: An End-to-End Benchmarking Framework for Compute-in-Memory Accelerators for On-chip Trainingcited by 33
  8. 2023X-Former: In-Memory Acceleration of Transformerscited by 25
  9. 2022Efficient Visual Tracking via Hierarchical Cross-Attention Transformercited by 24
  10. 2021PSViT: Better Vision Transformer via Token Pooling and Attention Sharingcited by 21
  11. 2018RxNN: A Framework for Evaluating Deep Neural Networks on Resistive Crossbarscited by 19
  12. 2020TIMELY: Pushing Data Movements and Interfaces in PIM Accelerators Towards Local and in Time Domaincited by 12
  13. 2022Examining the Robustness of Spiking Neural Networks on Non-ideal Memristive Crossbarscited by 9
  14. 2022SpikeSim: An end-to-end Compute-in-Memory Hardware Evaluation Tool for Benchmarking Spiking Neural Networkscited by 9
  15. 2023XPert: Peripheral Circuit & Neural Architecture Co-search for Area and Energy-efficient Xbar-based Computingcited by 6
  16. 2023Examining the Role and Limits of Batchnorm Optimization to Mitigate Diverse Hardware-noise in In-memory Computingcited by 5
  17. 2022SPIN: An Empirical Evaluation on Sharing Parameters of Isotropic Networkscited by 1

Built on by

Cited by year

Citing papers per year: 2024: 1; 2025: 1.

2 papers.

  1. 2024A Survey: Collaborative Hardware and Software Design in the Era of Large Language Modelscited by 23
  2. 2025MEADOW: Memory-efficient Dataflow and Data Packing for Low Power Edge LLMscited by 4

Similar, not cited

Close in wording, no citation either way

  1. 2023A survey on efficient vision transformers: algorithms, techniques, and performance benchmarking
  2. 2025EdgeFlex-Transformer: Transformer Inference for Edge Devices
  3. 2025Low-Bit Integerization of Vision Transformers using Operand Reordering for Efficient Hardware
  4. 2022Q-ViT: Accurate and Fully Quantized Low-bit Vision Transformer
  5. 2026QFlash: Bridging Quantization and Memory Efficiency in Vision Transformer Attention
  6. 2025LL-ViT: Edge Deployable Vision Transformers with Look Up Table Neurons
  7. 2026ToaSt: Token Channel Selection and Structured Pruning for Efficient ViT
  8. 2023Edge-MoE: Memory-Efficient Multi-Task Vision Transformer Architecture with Task-level Sparsity via Mixture-of-Experts