mlsanityData to 7 Oct 2026

Conditional Positional Encodings for Vision Transformers

arXiv 2102.10882 ↗First posted 22 Feb 2021cs.CV cs.AI cs.LGcited by 402

Not on the map. Why?

Abstract (arXiv)

We propose a conditional positional encoding (CPE) scheme for vision Transformers. Unlike previous fixed or learnable positional encodings, which are pre-defined and independent of input tokens, CPE is dynamically generated and conditioned on the local neighborhood of the input tokens. As a result, CPE can easily generalize to the input sequences that are longer than what the model has ever seen during training. Besides, CPE can keep the desired translation-invariance in the image classification task, resulting in improved performance. We implement CPE with a simple Position Encoding Generator (PEG) to get seamlessly incorporated into the current Transformer framework. Built on PEG, we present Conditional Position encoding Vision Transformer (CPVT). We demonstrate that CPVT has visually similar attention maps compared to those with learned positional encodings and delivers outperforming results. Our code is available at https://github.com/Meituan-AutoML/CPVT .

Key facts. Cited by 402 papers in the map (data through 7 Oct 2026). Builds on 39 of its 58 references that are in the map. First posted on arXiv 22 Feb 2021.

Builds on

39 of 58 references.

References · 39

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  6. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  7. 2020End-to-End Object Detection with Transformerscited by 7,384
  8. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,415
  9. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,451
  10. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,155
  11. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  12. 2020Deformable DETR: Deformable Transformers for End-to-End Object Detectioncited by 3,410
27 more
  1. 2019CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Featurescited by 2,875
  2. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,801
  3. 2019RandAugment: Practical automated data augmentation with a reduced search spacecited by 2,261
  4. 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,157
  5. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,838
  6. 2020Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformerscited by 1,368
  7. 2017Random Erasing Data Augmentationcited by 1,235
  8. 2018Self-Attention with Relative Position Representationscited by 1,112
  9. 2016Deep Networks with Stochastic Depthcited by 973
  10. 2020Designing Network Design Spacescited by 954
  11. 2017Revisiting Unreasonable Effectiveness of Data in Deep Learning Eracited by 916
  12. 2020Pre-Trained Image Processing Transformercited by 891
  13. 2018Image Transformercited by 770
  14. 2019Panoptic Feature Pyramid Networkscited by 644
  15. 2019Stand-Alone Self-Attention in Vision Modelscited by 542
  16. 2020End-to-End Video Instance Segmentation with Transformerscited by 443
  17. 2019Attention Augmented Convolutional Networkscited by 363
  18. 2020Are we done with ImageNet?cited by 329
  19. 2019On the Relationship between Self-Attention and Convolutional Layerscited by 299
  20. 2015MXNet: A Flexible and Efficient Machine Learning Library for Heterogeneous Distributed Systemscited by 195
  21. 2020How Much Position Information Do Convolutional Neural Networks Encode?cited by 171
  22. 2018QANet: Combining Local Convolution with Global Self-Attention for Reading Comprehensioncited by 164
  23. 2021LambdaNetworks: Modeling Long-Range Interactions Without Attentioncited by 107
  24. 2020Learning to Encode Position for Transformer with Continuous Dynamical Modelcited by 61
  25. 2019Encoding word order in complex embeddingscited by 53
  26. 2019Transformers with convolutional context for ASRcited by 44
  27. 2019Convolutional Self-Attention Networkscited by 32

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 54; 2022: 98; 2023: 70; 2024: 80; 2025: 58; 2026: 42.

402 papers.

Sort:

By area:

  1. 2021SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformerscited by 2,923
  2. 2021DynamicViT: Efficient Vision Transformers with Dynamic Token Sparsificationcited by 849
  3. 2021PVT v2: Improved Baselines with Pyramid Vision Transformercited by 749
  4. 2023Point Transformer V3: Simpler, Faster, Strongercited by 579
  5. 2021Twins: Revisiting the Design of Spatial Attention in Vision Transformerscited by 565
  6. 2022Vision Transformer Adapter for Dense Predictionscited by 564
  7. 2021TransMIL: Transformer based Correlated Multiple Instance Learning for Whole Slide Image Classificationcited by 549
  8. 2021CSWin Transformer: A General Vision Transformer Backbone with Cross-Shaped Windowscited by 526
  9. 2022MaxViT: Multi-Axis Vision Transformercited by 476
  10. 2021A Survey of Transformerscited by 399
  11. 2021CMT: Convolutional Neural Networks Meet Vision Transformerscited by 310
  12. 2021Focal Self-attention for Local-Global Interactions in Vision Transformerscited by 286
38 more
  1. 2022Transformers Learn Shortcuts to Automatacited by 260
  2. 2022TinyViT: Fast Pretraining Distillation for Small Vision Transformerscited by 232
  3. 2024Rotary Position Embedding for Vision Transformercited by 226
  4. 2021Co-Scale Conv-Attentional Image Transformerscited by 224
  5. 2021ViTAE: Vision Transformer Advanced by Exploring Intrinsic Inductive Biascited by 216
  6. 2021FaceFormer: Speech-Driven 3D Facial Animation with Transformerscited by 208
  7. 2022Medical Image Segmentation Review: The success of U-Netcited by 190
  8. 2024CBraMod: A Criss-Cross Brain Foundation Model for EEG Decodingcited by 188
  9. 2021Adaptive Token Sampling For Efficient Vision Transformerscited by 181
  10. 2022DaViT: Dual Attention Vision Transformerscited by 174
  11. 2024MobileVLM V2: Faster and Stronger Baseline for Vision Language Modelcited by 172
  12. 2023CROMA: Remote Sensing Representations with Contrastive Radar-Optical Masked Autoencoderscited by 167
  13. 2021SPViT: Enabling Faster Vision Transformers via Soft Token Pruningcited by 165
  14. 2023FastViT: A Fast Hybrid Vision Transformer using Structural Reparameterizationcited by 161
  15. 2022Recurrent Vision Transformers for Object Detection with Event Camerascited by 158
  16. 2022Scaling Up Your Kernels to 31x31: Revisiting Large Kernel Design in CNNscited by 155
  17. 2021Not All Images are Worth 16x16 Words: Dynamic Transformers for Efficient Image Recognitioncited by 143
  18. 2022Stripformer: Strip Transformer for Fast Image Deblurringcited by 139
  19. 2022Multi-class Token Transformer for Weakly Supervised Semantic Segmentationcited by 138
  20. 2021SOFT: Softmax-free Transformer with Linear Complexitycited by 128
  21. 2021RegionViT: Regional-to-Local Attention for Vision Transformerscited by 126
  22. 2023MobileVLM : A Fast, Strong and Open Vision Language Assistant for Mobile Devicescited by 120
  23. 2021A Survey of Visual Transformerscited by 117
  24. 2022EdgeViTs: Competing Light-weight CNNs on Mobile Devices with Vision Transformerscited by 116
  25. 2021Shuffle Transformer: Rethinking Spatial Shuffle for Vision Transformercited by 115
  26. 2022MonoDTR: Monocular 3D Object Detection with Depth-Aware Transformercited by 108
  27. 2022ConvMAE: Masked Convolution Meets Masked Autoencoderscited by 105
  28. 2023OctFormer: Octree-based Transformers for 3D Point Cloudscited by 105
  29. 2021MISSFormer: An Effective Medical Image Segmentation Transformercited by 103
  30. 2021AS-MLP: An Axial Shifted MLP Architecture for Visioncited by 101
  31. 2024Demystify Mamba in Vision: A Linear Attention Perspectivecited by 101
  32. 2024TimeSuite: Improving MLLMs for Long Video Understanding via Grounded Tuningcited by 99
  33. 2022Global Context Vision Transformerscited by 98
  34. 2023Extracting Motion and Appearance via Inter-Frame Attention for Efficient Video Frame Interpolationcited by 97
  35. 2022Cross Aggregation Transformer for Image Restorationcited by 92
  36. 2024MambaIRv2: Attentive State Space Restorationcited by 90
  37. 2021Scalable Vision Transformers with Hierarchical Poolingcited by 84
  38. 2021P2T: Pyramid Pooling Transformer for Scene Understandingcited by 84

50 most cited of 402 All 402 (JSON)