# Feature Pyramid Transformer

arXiv: https://arxiv.org/abs/2007.09451 · HTML: https://mlsanity.com/paper/2007.09451/

## Key facts

- Cited by 96 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 18 Jul 2020.

## Built on by (most cited first, 30 of 96)

- [MOTR: End-to-End Multiple-Object Tracking with Transformer](https://mlsanity.com/paper/2105.03247/) (2021)
- [Remote Sensing Image Change Detection with Transformers](https://mlsanity.com/paper/2103.00208/) (2021)
- [Anti-Oversmoothing in Deep Vision Transformers via the Fourier Domain Analysis: From Theory to Practice](https://mlsanity.com/paper/2203.05962/) (2022)
- [Transformer-Based Visual Segmentation: A Survey](https://mlsanity.com/paper/2304.09854/) (2023)
- [A Survey of Visual Transformers](https://mlsanity.com/paper/2111.06091/) (2021)
- [On the Integration of Self-Attention and Convolution](https://mlsanity.com/paper/2111.14556/) (2021)
- [Representation Compensation Networks for Continual Semantic Segmentation](https://mlsanity.com/paper/2203.05402/) (2022)
- [Efficient Video Transformers with Spatial-Temporal Token Selection](https://mlsanity.com/paper/2111.11591/) (2021)
- [Medical Image Segmentation Using Squeeze-and-Expansion Transformers](https://mlsanity.com/paper/2105.09511/) (2021)
- [Singular Value Fine-tuning: Few-shot Segmentation requires Few-parameters Fine-tuning](https://mlsanity.com/paper/2206.06122/) (2022)
- [A Large-Scale Benchmark for Food Image Segmentation](https://mlsanity.com/paper/2105.05409/) (2021)
- [HiCLIP: Contrastive Language-Image Pretraining with Hierarchy-aware Attention](https://mlsanity.com/paper/2303.02995/) (2023)
- [CTNet: Context-based Tandem Network for Semantic Segmentation](https://mlsanity.com/paper/2104.09805/) (2021)
- [TSA-Net: Tube Self-Attention Network for Action Quality Assessment](https://mlsanity.com/paper/2201.03746/) (2022)
- [Spatial-Spectral Transformer for Hyperspectral Image Denoising](https://mlsanity.com/paper/2211.14090/) (2022)
- [Dual-stream Network for Visual Recognition](https://mlsanity.com/paper/2105.14734/) (2021)
- [Multi-scale Matching Networks for Semantic Correspondence](https://mlsanity.com/paper/2108.00211/) (2021)
- [StructToken : Rethinking Semantic Segmentation with Structural Prior](https://mlsanity.com/paper/2203.12612/) (2022)
- [PointFlow: Flowing Semantics Through Points for Aerial Image Segmentation](https://mlsanity.com/paper/2103.06564/) (2021)
- [Meta-attention for ViT-backed Continual Learning](https://mlsanity.com/paper/2203.11684/) (2022)
- [InstaFormer: Instance-Aware Image-to-Image Translation with Transformer](https://mlsanity.com/paper/2203.16248/) (2022)
- [GraphFPN: Graph Feature Pyramid Network for Object Detection](https://mlsanity.com/paper/2108.00580/) (2021)
- [Dense Interaction Learning for Video-based Person Re-identification](https://mlsanity.com/paper/2103.09013/) (2021)
- [Fully Transformer Networks for Semantic Image Segmentation](https://mlsanity.com/paper/2106.04108/) (2021)
- [Self-Regulation for Semantic Segmentation](https://mlsanity.com/paper/2108.09702/) (2021)
- [AFPN: Asymptotic Feature Pyramid Network for Object Detection](https://mlsanity.com/paper/2306.15988/) (2023)
- [Quantization Variation: A New Perspective on Training Transformers with Low-Bit Precision](https://mlsanity.com/paper/2307.00331/) (2023)
- [Tri2-plane: Thinking Head Avatar via Feature Pyramid](https://mlsanity.com/paper/2401.09386/) (2024)
- [Point Cloud Learning with Transformer](https://mlsanity.com/paper/2104.13636/) (2021)
- [Efficient and Accurate Multi-scale Topological Network for Single Image Dehazing](https://mlsanity.com/paper/2102.12135/) (2021)

## Abstract (arXiv)

Feature interactions across space and scales underpin modern visual recognition systems because they introduce beneficial visual contexts. Conventionally, spatial contexts are passively hidden in the CNN's increasing receptive fields or actively encoded by non-local convolution. Yet, the non-local spatial interactions are not across scales, and thus they fail to capture the non-local contexts of objects (or parts) residing in different scales. To this end, we propose a fully active feature interaction across both space and scales, called Feature Pyramid Transformer (FPT). It transforms any feature pyramid into another feature pyramid of the same size but with richer contexts, by using three specially designed transformers in self-level, top-down, and bottom-up interaction fashion. FPT serves as a generic visual backbone with fair computational overhead. We conduct extensive experiments in both instance-level (i.e., object detection and instance segmentation) and pixel-level segmentation tasks, using various backbones and head networks, and observe consistent improvement over all the baselines and the state-of-the-art methods.