# MSG-Transformer: Exchanging Local Spatial Information by Manipulating Messenger Tokens

arXiv: https://arxiv.org/abs/2105.15168 · HTML: https://mlsanity.com/paper/2105.15168/

## Key facts

- Cited by 46 papers in the map (data through 14 Oct 2026).
- Builds on 53 of its 63 references that are in the map.
- First posted on arXiv 31 May 2021.

## Builds on (most cited first, 30 of 53)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016)
- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- [Rethinking the Inception Architecture for Computer Vision](https://mlsanity.com/paper/1512.00567/) (2015)
- [SGDR: Stochastic Gradient Descent with Warm Restarts](https://mlsanity.com/paper/1608.03983/) (2016)
- [You Only Look Once: Unified, Real-Time Object Detection](https://mlsanity.com/paper/1506.02640/) (2015)
- [Training data-efficient image transformers & distillation through attention](https://mlsanity.com/paper/2012.12877/) (2020)
- [Going Deeper with Convolutions](https://mlsanity.com/paper/1409.4842/) (2014)
- [MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications](https://mlsanity.com/paper/1704.04861/) (2017)
- [DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs](https://mlsanity.com/paper/1606.00915/) (2016)
- [SSD: Single Shot MultiBox Detector](https://mlsanity.com/paper/1512.02325/) (2015)
- [Aggregated Residual Transformations for Deep Neural Networks](https://mlsanity.com/paper/1611.05431/) (2016)
- [Xception: Deep Learning with Depthwise Separable Convolutions](https://mlsanity.com/paper/1610.02357/) (2016)
- [Non-local Neural Networks](https://mlsanity.com/paper/1711.07971/) (2017)
- [Rethinking Atrous Convolution for Semantic Image Segmentation](https://mlsanity.com/paper/1706.05587/) (2017)
- [Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutions](https://mlsanity.com/paper/2102.12122/) (2021)
- [MMDetection: Open MMLab Detection Toolbox and Benchmark](https://mlsanity.com/paper/1906.07155/) (2019)
- [Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformers](https://mlsanity.com/paper/2012.15840/) (2020)

## Built on by (most cited first, 30 of 46)

- [Vision Mamba: Efficient Visual Representation Learning with Bidirectional State Space Model](https://mlsanity.com/paper/2401.09417/) (2024)
- [You Only Look at One Sequence: Rethinking Transformer in Vision through Object Detection](https://mlsanity.com/paper/2106.00666/) (2021)
- [HiFormer: Hierarchical Multi-scale Representations Using Transformers for Medical Image Segmentation](https://mlsanity.com/paper/2207.08518/) (2022)
- [CF-ViT: A General Coarse-to-Fine Method for Vision Transformer](https://mlsanity.com/paper/2203.03821/) (2022)
- [Temporally Efficient Vision Transformer for Video Instance Segmentation](https://mlsanity.com/paper/2204.08412/) (2022)
- [Vision Transformer with Quadrangle Attention](https://mlsanity.com/paper/2303.15105/) (2023)
- [A Survey on Transformer Compression](https://mlsanity.com/paper/2402.05964/) (2024)
- [MixFormer: Mixing Features across Windows and Dimensions](https://mlsanity.com/paper/2204.02557/) (2022)
- [VSA: Learning Varied-Size Window Attention in Vision Transformers](https://mlsanity.com/paper/2204.08446/) (2022)
- [MAE-DFER: Efficient Masked Autoencoder for Self-supervised Dynamic Facial Expression Recognition](https://mlsanity.com/paper/2307.02227/) (2023)
- [LightViT: Towards Light-Weight Convolution-Free Vision Transformers](https://mlsanity.com/paper/2207.05557/) (2022)
- [Sparse MLP for Image Recognition: Is Self-Attention Really Necessary?](https://mlsanity.com/paper/2109.05422/) (2021)
- [Pale Transformer: A General Vision Transformer Backbone with Pale-Shaped Attention](https://mlsanity.com/paper/2112.14000/) (2021)
- [Towards Data-Efficient Detection Transformers](https://mlsanity.com/paper/2203.09507/) (2022)
- [Scaled ReLU Matters for Training Vision Transformers](https://mlsanity.com/paper/2109.03810/) (2021)
- [Bridging the Gap Between Vision Transformers and Convolutional Neural Networks on Small Datasets](https://mlsanity.com/paper/2210.05958/) (2022)
- [TransFace++: Rethinking the Face Recognition Paradigm with a Focus on Accuracy, Efficiency, and Security](https://mlsanity.com/paper/2308.10133/) (2023)
- [SCTNet: Single-Branch CNN with Transformer Semantic Information for Real-Time Segmentation](https://mlsanity.com/paper/2312.17071/) (2023)
- [DAT++: Spatially Dynamic Vision Transformer with Deformable Attention](https://mlsanity.com/paper/2309.01430/) (2023)
- [Robustifying Token Attention for Vision Transformers](https://mlsanity.com/paper/2303.11126/) (2023)
- [Dynamic Group Transformer: A General Vision Transformer Backbone with Dynamic Group Attention](https://mlsanity.com/paper/2203.03937/) (2022)
- [Beyond Masking: Demystifying Token-Based Pre-Training for Vision Transformers](https://mlsanity.com/paper/2203.14313/) (2022)
- [Computation-efficient Deep Learning for Computer Vision: A Survey](https://mlsanity.com/paper/2308.13998/) (2023)
- [Depth-Wise Convolutions in Vision Transformers for Efficient Training on Small Datasets](https://mlsanity.com/paper/2407.19394/) (2024)
- [A Lightweight Sparse Focus Transformer for Remote Sensing Image Change Captioning](https://mlsanity.com/paper/2405.06598/) (2024)
- [ViG: Linear-complexity Visual Sequence Learning with Gated Linear Attention](https://mlsanity.com/paper/2405.18425/) (2024)
- [Tracking Instances as Queries](https://mlsanity.com/paper/2106.11963/) (2021)
- [TransPillars: Coarse-to-Fine Aggregation for Multi-Frame 3D Object Detection](https://mlsanity.com/paper/2208.03141/) (2022)
- [DQnet: Cross-Model Detail Querying for Camouflaged Object Detection](https://mlsanity.com/paper/2212.08296/) (2022)
- [MobileInst: Video Instance Segmentation on the Mobile](https://mlsanity.com/paper/2303.17594/) (2023)

## Abstract (arXiv)

Transformers have offered a new methodology of designing neural networks for visual recognition. Compared to convolutional networks, Transformers enjoy the ability of referring to global features at each stage, yet the attention module brings higher computational overhead that obstructs the application of Transformers to process high-resolution visual data. This paper aims to alleviate the conflict between efficiency and flexibility, for which we propose a specialized token for each region that serves as a messenger (MSG). Hence, by manipulating these MSG tokens, one can flexibly exchange visual information across regions and the computational complexity is reduced. We then integrate the MSG token into a multi-scale architecture named MSG-Transformer. In standard image classification and object detection, MSG-Transformer achieves competitive performance and the inference on both GPU and CPU is accelerated. Code is available at https://github.com/hustvl/MSG-Transformer.