# Recurrent Glimpse-based Decoder for Detection with Transformer

arXiv: https://arxiv.org/abs/2112.04632 · HTML: https://mlsanity.com/paper/2112.04632/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- Builds on 39 of its 49 references that are in the map.
- First posted on arXiv 9 Dec 2021.

## Builds on (most cited first, 30 of 39)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [Feature Pyramid Networks for Object Detection](https://mlsanity.com/paper/1612.03144/) (2016)
- [You Only Look Once: Unified, Real-Time Object Detection](https://mlsanity.com/paper/1506.02640/) (2015)
- [Squeeze-and-Excitation Networks](https://mlsanity.com/paper/1709.01507/) (2017)
- [Training data-efficient image transformers & distillation through attention](https://mlsanity.com/paper/2012.12877/) (2020)
- [Deformable DETR: Deformable Transformers for End-to-End Object Detection](https://mlsanity.com/paper/2010.04159/) (2020)
- [SSD: Single Shot MultiBox Detector](https://mlsanity.com/paper/1512.02325/) (2015)
- [Fast R-CNN](https://mlsanity.com/paper/1504.08083/) (2015)
- [Aggregated Residual Transformations for Deep Neural Networks](https://mlsanity.com/paper/1611.05431/) (2016)
- [RAFT: Recurrent All-Pairs Field Transforms for Optical Flow](https://mlsanity.com/paper/2003.12039/) (2020)
- [Rich feature hierarchies for accurate object detection and semantic segmentation](https://mlsanity.com/paper/1311.2524/) (2013)
- [FCOS: Fully Convolutional One-Stage Object Detection](https://mlsanity.com/paper/1904.01355/) (2019)
- [Deformable Convolutional Networks](https://mlsanity.com/paper/1703.06211/) (2017)
- [Cascade R-CNN: Delving into High Quality Object Detection](https://mlsanity.com/paper/1712.00726/) (2017)
- [Efficient Transformers: A Survey](https://mlsanity.com/paper/2009.06732/) (2020)
- [CornerNet: Detecting Objects as Paired Keypoints](https://mlsanity.com/paper/1808.01244/) (2018)
- [Stand-Alone Self-Attention in Vision Models](https://mlsanity.com/paper/1906.05909/) (2019)
- [Sparse R-CNN: End-to-End Object Detection with Learnable Proposals](https://mlsanity.com/paper/2011.12450/) (2020)
- [Hybrid Task Cascade for Instance Segmentation](https://mlsanity.com/paper/1901.07518/) (2019)
- [ResNeSt: Split-Attention Networks](https://mlsanity.com/paper/2004.08955/) (2020)
- [Conditional DETR for Fast Training Convergence](https://mlsanity.com/paper/2108.06152/) (2021)

## Built on by (most cited first, 21 of 21)

- [An Empirical Study of Remote Sensing Pretraining](https://mlsanity.com/paper/2204.02825/) (2022)
- [Towards Data-Efficient Detection Transformers](https://mlsanity.com/paper/2203.09507/) (2022)
- [Object Detection with Transformers: A Review](https://mlsanity.com/paper/2306.04670/) (2023)
- [Cascade-DETR: Delving into High-Quality Universal Object Detection](https://mlsanity.com/paper/2307.11035/) (2023)
- [Pose-disentangled Contrastive Learning for Self-supervised Facial Representation](https://mlsanity.com/paper/2211.13490/) (2022)
- [Centralized Feature Pyramid for Object Detection](https://mlsanity.com/paper/2210.02093/) (2022)
- [Sparse Semi-DETR: Sparse Learnable Queries for Semi-Supervised Object Detection](https://mlsanity.com/paper/2404.01819/) (2024)
- [Deep Equilibrium Object Detection](https://mlsanity.com/paper/2308.09564/) (2023)
- [StageInteractor: Query-based Object Detector with Cross-stage Interaction](https://mlsanity.com/paper/2304.04978/) (2023)
- [ASAG: Building Strong One-Decoder-Layer Sparse Detectors via Adaptive Sparse Anchor Generation](https://mlsanity.com/paper/2308.09242/) (2023)
- [A Hybrid Approach for Document Layout Analysis in Document images](https://mlsanity.com/paper/2404.17888/) (2024)
- [Transformer-based Context Condensation for Boosting Feature Pyramids in Object Detection](https://mlsanity.com/paper/2207.06603/) (2022)
- [Towards End-to-End Semi-Supervised Table Detection with Semantic Aligned Matching Transformer](https://mlsanity.com/paper/2405.00187/) (2024)
- [HVPNet: A Bio-Inspired Network for General Salient and Camouflaged Object Detection](https://mlsanity.com/paper/2606.31496/) (2026)
- [RecursiveDet: End-to-End Region-based Recursive Object Detection](https://mlsanity.com/paper/2307.13619/) (2023)
- [Box-DETR: Understanding and Boxing Conditional Spatial Queries](https://mlsanity.com/paper/2307.08353/) (2023)
- [Position Embedding Needs an Independent Layer Normalization](https://mlsanity.com/paper/2212.05262/) (2022)
- [SSGA-Net: Stepwise Spatial Global-local Aggregation Networks for for Autonomous Driving](https://mlsanity.com/paper/2405.18857/) (2024)
- [Mr. DETR++: Instructive Multi-Route Training for Detection Transformers with Mixture-of-Experts](https://mlsanity.com/paper/2412.10028/) (2024)
- [Few-Shot Object Detection via Spatial-Channel State Space Model](https://mlsanity.com/paper/2507.15308/) (2025)
- [Heuristic-inspired Reasoning Priors Facilitate Data-Efficient Referring Object Detection](https://mlsanity.com/paper/2603.24166/) (2026)

## Abstract (arXiv)

Although detection with Transformer (DETR) is increasingly popular, its global attention modeling requires an extremely long training period to optimize and achieve promising detection performance. Alternative to existing studies that mainly develop advanced feature or embedding designs to tackle the training issue, we point out that the Region-of-Interest (RoI) based detection refinement can easily help mitigate the difficulty of training for DETR methods. Based on this, we introduce a novel REcurrent Glimpse-based decOder (REGO) in this paper. In particular, the REGO employs a multi-stage recurrent processing structure to help the attention of DETR gradually focus on foreground objects more accurately. In each processing stage, visual features are extracted as glimpse features from RoIs with enlarged bounding box areas of detection results from the previous stage. Then, a glimpse-based decoder is introduced to provide refined detection results based on both the glimpse features and the attention modeling outputs of the previous stage. In practice, REGO can be easily embedded in representative DETR variants while maintaining their fully end-to-end training and inference pipelines. In particular, REGO helps Deformable DETR achieve 44.8 AP on the MSCOCO dataset with only 36 training epochs, compared with the first DETR and the Deformable DETR that require 500 and 50 epochs to achieve comparable performance, respectively. Experiments also show that REGO consistently boosts the performance of different DETR detectors by up to 7% relative gain at the same setting of 50 training epochs. Code is available via https://github.com/zhechen/Deformable-DETR-REGO.