# Dual Vision Transformer

arXiv: https://arxiv.org/abs/2207.04976 · HTML: https://mlsanity.com/paper/2207.04976/

## Key facts

- Cited by 29 papers in the map (data through 14 Oct 2026).
- Builds on 61 of its 70 references that are in the map.
- First posted on arXiv 11 Jul 2022.

## Builds on (most cited first, 30 of 61)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- [Rethinking the Inception Architecture for Computer Vision](https://mlsanity.com/paper/1512.00567/) (2015)
- [SGDR: Stochastic Gradient Descent with Warm Restarts](https://mlsanity.com/paper/1608.03983/) (2016)
- [Training data-efficient image transformers & distillation through attention](https://mlsanity.com/paper/2012.12877/) (2020)
- [Going Deeper with Convolutions](https://mlsanity.com/paper/1409.4842/) (2014)
- [Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation](https://mlsanity.com/paper/1802.02611/) (2018)
- [Aggregated Residual Transformations for Deep Neural Networks](https://mlsanity.com/paper/1611.05431/) (2016)
- [RandAugment: Practical automated data augmentation with a reduced search space](https://mlsanity.com/paper/1909.13719/) (2019)
- [Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutions](https://mlsanity.com/paper/2102.12122/) (2021)
- [Do ImageNet Classifiers Generalize to ImageNet?](https://mlsanity.com/paper/1902.10811/) (2019)
- [Deep High-Resolution Representation Learning for Visual Recognition](https://mlsanity.com/paper/1908.07919/) (2019)
- [Cascade R-CNN: Delving into High Quality Object Detection](https://mlsanity.com/paper/1712.00726/) (2017)
- [Unified Perceptual Parsing for Scene Understanding](https://mlsanity.com/paper/1807.10221/) (2018)
- [Random Erasing Data Augmentation](https://mlsanity.com/paper/1708.04896/) (2017)
- [Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNet](https://mlsanity.com/paper/2101.11986/) (2021)
- [Semantic Understanding of Scenes through the ADE20K Dataset](https://mlsanity.com/paper/1608.05442/) (2016)
- [CvT: Introducing Convolutions to Vision Transformers](https://mlsanity.com/paper/2103.15808/) (2021)
- [Multiscale Vision Transformers](https://mlsanity.com/paper/2104.11227/) (2021)
- [CrossViT: Cross-Attention Multi-Scale Vision Transformer for Image Classification](https://mlsanity.com/paper/2103.14899/) (2021)
- [Going deeper with Image Transformers](https://mlsanity.com/paper/2103.17239/) (2021)
- [PVT v2: Improved Baselines with Pyramid Vision Transformer](https://mlsanity.com/paper/2106.13797/) (2021)
- [A Survey on Visual Transformer](https://mlsanity.com/paper/2012.12556/) (2020)
- [Panoptic Feature Pyramid Networks](https://mlsanity.com/paper/1901.02446/) (2019)

## Built on by (most cited first, 29 of 29)

- [MetaFormer Baselines for Vision](https://mlsanity.com/paper/2210.13452/) (2022)
- [RMT: Retentive Networks Meet Vision Transformers](https://mlsanity.com/paper/2309.11523/) (2023)
- [VideoStudio: Generating Consistent-Content and Multi-Scene Videos](https://mlsanity.com/paper/2401.01256/) (2024)
- [Advances in Medical Image Analysis with Vision Transformers: A Comprehensive Review](https://mlsanity.com/paper/2301.03505/) (2023)
- [SD-DiT: Unleashing the Power of Self-supervised Discrimination in Diffusion Transformer](https://mlsanity.com/paper/2403.17004/) (2024)
- [Hierarchical Local-Global Transformer for Temporal Sentence Grounding](https://mlsanity.com/paper/2208.14882/) (2022)
- [Jigsaw-ViT: Learning Jigsaw Puzzles in Vision Transformer](https://mlsanity.com/paper/2207.11971/) (2022)
- [Deep Convolutional Pooling Transformer for Deepfake Detection](https://mlsanity.com/paper/2209.05299/) (2022)
- [LSNet: See Large, Focus Small](https://mlsanity.com/paper/2503.23135/) (2025)
- [HIRI-ViT: Scaling Vision Transformer with High Resolution Inputs](https://mlsanity.com/paper/2403.11999/) (2024)
- [Improving Text-guided Object Inpainting with Semantic Pre-inpainting](https://mlsanity.com/paper/2409.08260/) (2024)
- [Perceiving Longer Sequences With Bi-Directional Cross-Attention Transformers](https://mlsanity.com/paper/2402.12138/) (2024)
- [When Pedestrian Detection Meets Multi-Modal Learning: Generalist Model and Benchmark Dataset](https://mlsanity.com/paper/2407.10125/) (2024)
- [Resolution Enhancement Processing on Low Quality Images Using Swin Transformer Based on Interval Dense Connection Strategy](https://mlsanity.com/paper/2303.09190/) (2023)
- [CAIT: Triple-Win Compression towards High Accuracy, Fast Inference, and Favorable Transferability For ViTs](https://mlsanity.com/paper/2309.15755/) (2023)
- [Dual-Scale Transformer for Large-Scale Single-Pixel Imaging](https://mlsanity.com/paper/2404.05001/) (2024)
- [Efficient Partitioning Vision Transformer on Edge Devices for Distributed Inference](https://mlsanity.com/paper/2410.11650/) (2024)
- [Most Important Person-guided Dual-branch Cross-Patch Attention for Group Affect Recognition](https://mlsanity.com/paper/2212.07055/) (2022)
- [HAFormer: Unleashing the Power of Hierarchy-Aware Features for Lightweight Semantic Segmentation](https://mlsanity.com/paper/2407.07441/) (2024)
- [ReViT: Enhancing Vision Transformers Feature Diversity with Attention Residual Connections](https://mlsanity.com/paper/2402.11301/) (2024)
- [Innovative Tooth Segmentation Using Hierarchical Features and Bidirectional Sequence Modeling](https://mlsanity.com/paper/2602.21712/) (2026)
- [DualToken-ViT: Position-aware Efficient Vision Transformer with Dual Token Fusion](https://mlsanity.com/paper/2309.12424/) (2023)
- [EViT: An Eagle Vision Transformer with Bi-Fovea Self-Attention](https://mlsanity.com/paper/2310.06629/) (2023)
- [Edge Attention Module for Object Classification](https://mlsanity.com/paper/2502.03103/) (2025)
- [Hyb-KAN ViT: Hybrid Kolmogorov-Arnold Networks Augmented Vision Transformer](https://mlsanity.com/paper/2505.04740/) (2025)
- [Tri-FusionNet: Enhancing Image Description Generation with Transformer-based Fusion Network and Dual Attention Mechanism](https://mlsanity.com/paper/2504.16761/) (2025)
- [Advanced Chest X-Ray Analysis via Transformer-Based Image Descriptors and Cross-Model Attention Mechanism](https://mlsanity.com/paper/2504.16774/) (2025)
- [Random Wins All: Rethinking Grouping Strategies for Vision Tokens](https://mlsanity.com/paper/2603.00486/) (2026)
- [What-Where Transformer: A Slot-Centric Visual Backbone for Concurrent Representation and Localization](https://mlsanity.com/paper/2605.12021/) (2026)

## Abstract (arXiv)

Prior works have proposed several strategies to reduce the computational cost of self-attention mechanism. Many of these works consider decomposing the self-attention procedure into regional and local feature extraction procedures that each incurs a much smaller computational complexity. However, regional information is typically only achieved at the expense of undesirable information lost owing to down-sampling. In this paper, we propose a novel Transformer architecture that aims to mitigate the cost issue, named Dual Vision Transformer (Dual-ViT). The new architecture incorporates a critical semantic pathway that can more efficiently compress token vectors into global semantics with reduced order of complexity. Such compressed global semantics then serve as useful prior information in learning finer pixel level details, through another constructed pixel pathway. The semantic pathway and pixel pathway are then integrated together and are jointly trained, spreading the enhanced self-attention information in parallel through both of the pathways. Dual-ViT is henceforth able to reduce the computational complexity without compromising much accuracy. We empirically demonstrate that Dual-ViT provides superior accuracy than SOTA Transformer architectures with reduced training complexity. Source code is available at urlhttps://github.com/YehLi/ImageNetModel.