# Towards Data-Efficient Detection Transformers

arXiv: https://arxiv.org/abs/2203.09507 · HTML: https://mlsanity.com/paper/2203.09507/

## Key facts

- Cited by 34 papers in the map (data through 14 Oct 2026).
- Builds on 51 of its 61 references that are in the map.
- First posted on arXiv 17 Mar 2022.

## Builds on (most cited first, 30 of 51)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [Feature Pyramid Networks for Object Detection](https://mlsanity.com/paper/1612.03144/) (2016)
- [You Only Look Once: Unified, Real-Time Object Detection](https://mlsanity.com/paper/1506.02640/) (2015)
- [Training data-efficient image transformers & distillation through attention](https://mlsanity.com/paper/2012.12877/) (2020)
- [The Cityscapes Dataset for Semantic Urban Scene Understanding](https://mlsanity.com/paper/1604.01685/) (2016)
- [Deformable DETR: Deformable Transformers for End-to-End Object Detection](https://mlsanity.com/paper/2010.04159/) (2020)
- [SSD: Single Shot MultiBox Detector](https://mlsanity.com/paper/1512.02325/) (2015)
- [SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformers](https://mlsanity.com/paper/2105.15203/) (2021)
- [Rich feature hierarchies for accurate object detection and semantic segmentation](https://mlsanity.com/paper/1311.2524/) (2013)
- [Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutions](https://mlsanity.com/paper/2102.12122/) (2021)
- [FCOS: Fully Convolutional One-Stage Object Detection](https://mlsanity.com/paper/1904.01355/) (2019)
- [Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformers](https://mlsanity.com/paper/2012.15840/) (2020)
- [Cascade R-CNN: Delving into High Quality Object Detection](https://mlsanity.com/paper/1712.00726/) (2017)
- [Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNet](https://mlsanity.com/paper/2101.11986/) (2021)
- [Data-Efficient Image Recognition with Contrastive Predictive Coding](https://mlsanity.com/paper/1905.09272/) (2019)
- [Transformer in Transformer](https://mlsanity.com/paper/2103.00112/) (2021)
- [Twins: Revisiting the Design of Spatial Attention in Vision Transformers](https://mlsanity.com/paper/2104.13840/) (2021)
- [Bridging the Gap Between Anchor-based and Anchor-free Detection via Adaptive Training Sample Selection](https://mlsanity.com/paper/1912.02424/) (2019)
- [Sparse R-CNN: End-to-End Object Detection with Learnable Proposals](https://mlsanity.com/paper/2011.12450/) (2020)
- [End-to-End Video Instance Segmentation with Transformers](https://mlsanity.com/paper/2011.14503/) (2020)
- [Conditional DETR for Fast Training Convergence](https://mlsanity.com/paper/2108.06152/) (2021)
- [Soft-NMS -- Improving Object Detection With One Line of Code](https://mlsanity.com/paper/1704.04503/) (2017)
- [Deeply-Supervised Nets](https://mlsanity.com/paper/1409.5185/) (2014)

## Built on by (most cited first, 30 of 34)

- [DETRs with Hybrid Matching](https://mlsanity.com/paper/2207.13080/) (2022)
- [Transformer-Based Visual Segmentation: A Survey](https://mlsanity.com/paper/2304.09854/) (2023)
- [DPText-DETR: Towards Better Scene Text Detection with Dynamic Points in Transformer](https://mlsanity.com/paper/2207.04491/) (2022)
- [RefSAM: Efficiently Adapting Segmenting Anything Model for Referring Video Object Segmentation](https://mlsanity.com/paper/2307.00997/) (2023)
- [GLT-T: Global-Local Transformer Voting for 3D Single Object Tracking in Point Clouds](https://mlsanity.com/paper/2211.10927/) (2022)
- [Deep Graph Reprogramming](https://mlsanity.com/paper/2304.14593/) (2023)
- [Cascade-DETR: Delving into High-Quality Universal Object Detection](https://mlsanity.com/paper/2307.11035/) (2023)
- [Improving Transferability for Domain Adaptive Detection Transformers](https://mlsanity.com/paper/2204.14195/) (2022)
- [Dynamic Focus-aware Positional Queries for Semantic Segmentation](https://mlsanity.com/paper/2204.01244/) (2022)
- [MS-DETR: Multispectral Pedestrian Detection Transformer with Loosely Coupled Fusion and Modality-Balanced Optimization](https://mlsanity.com/paper/2302.00290/) (2023)
- [Towards Efficient Use of Multi-Scale Features in Transformer-Based Object Detectors](https://mlsanity.com/paper/2208.11356/) (2022)
- [MDViT: Multi-domain Vision Transformer for Small Medical Image Segmentation Datasets](https://mlsanity.com/paper/2307.02100/) (2023)
- [MI-DETR: An Object Detection Model with Multi-time Inquiries Mechanism](https://mlsanity.com/paper/2503.01463/) (2025)
- [StageInteractor: Query-based Object Detector with Cross-stage Interaction](https://mlsanity.com/paper/2304.04978/) (2023)
- [Actor-agnostic Multi-label Action Recognition with Multi-modal Query](https://mlsanity.com/paper/2307.10763/) (2023)
- [DeepSolo++: Let Transformer Decoder with Explicit Points Solo for Multilingual Text Spotting](https://mlsanity.com/paper/2305.19957/) (2023)
- [Are Vision Transformers More Data Hungry Than Newborn Visual Systems?](https://mlsanity.com/paper/2312.02843/) (2023)
- [PlaneRecTR++: Unified Query Learning for Joint 3D Planar Reconstruction and Pose Estimation](https://mlsanity.com/paper/2307.13756/) (2023)
- [Transformer-based Context Condensation for Boosting Feature Pyramids in Object Detection](https://mlsanity.com/paper/2207.06603/) (2022)
- [GLT-T++: Global-Local Transformer for 3D Siamese Tracking with Ranking Loss](https://mlsanity.com/paper/2304.00242/) (2023)
- [YOLOPose V2: Understanding and Improving Transformer-based 6D Pose Estimation](https://mlsanity.com/paper/2307.11550/) (2023)
- [SamLP: A Customized Segment Anything Model for License Plate Detection](https://mlsanity.com/paper/2401.06374/) (2024)
- [OD3: Optimization-free Dataset Distillation for Object Detection](https://mlsanity.com/paper/2506.01942/) (2025)
- [OmniForce: On Human-Centered, Large Model Empowered and Cloud-Edge Collaborative AutoML System](https://mlsanity.com/paper/2303.00501/) (2023)
- [A Simple and Robust Framework for Cross-Modality Medical Image Segmentation applied to Vision Transformers](https://mlsanity.com/paper/2310.05572/) (2023)
- [FastTextSpotter: A High-Efficiency Transformer for Multilingual Scene Text Spotting](https://mlsanity.com/paper/2408.14998/) (2024)
- [Benchmarking individual tree segmentation using multispectral airborne laser scanning data: the FGI-EMIT dataset](https://mlsanity.com/paper/2511.00653/) (2025)
- [Thinking Twice: Clinical-Inspired Thyroid Ultrasound Lesion Detection Based on Feature Feedback](https://mlsanity.com/paper/2305.15114/) (2023)
- [Fisher-aware Quantization for DETR Detectors with Critical-category Objectives](https://mlsanity.com/paper/2407.03442/) (2024)
- [MDS-DETR: DETR with Masked Duplicate Suppressor](https://mlsanity.com/paper/2605.23507/) (2026)

## Abstract (arXiv)

Detection Transformers have achieved competitive performance on the sample-rich COCO dataset. However, we show most of them suffer from significant performance drops on small-size datasets, like Cityscapes. In other words, the detection transformers are generally data-hungry. To tackle this problem, we empirically analyze the factors that affect data efficiency, through a step-by-step transition from a data-efficient RCNN variant to the representative DETR. The empirical results suggest that sparse feature sampling from local image areas holds the key. Based on this observation, we alleviate the data-hungry issue of existing detection transformers by simply alternating how key and value sequences are constructed in the cross-attention layer, with minimum modifications to the original models. Besides, we introduce a simple yet effective label augmentation method to provide richer supervision and improve data efficiency. Experiments show that our method can be readily applied to different detection transformers and improve their performance on both small-size and sample-rich datasets. Code will be made publicly available at urlhttps://github.com/encounter1997/DE-DETRs.