# Less is More: Focus Attention for Efficient DETR

arXiv: https://arxiv.org/abs/2307.12612 · HTML: https://mlsanity.com/paper/2307.12612/

## Key facts

- Cited by 38 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 38 references that are in the map.
- First posted on arXiv 24 Jul 2023.

## Builds on (most cited first, 29 of 29)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Deformable DETR: Deformable Transformers for End-to-End Object Detection](https://mlsanity.com/paper/2010.04159/) (2020)
- [DINO: DETR with Improved DeNoising Anchor Boxes for End-to-End Object Detection](https://mlsanity.com/paper/2203.03605/) (2022)
- [DynamicViT: Efficient Vision Transformers with Dynamic Token Sparsification](https://mlsanity.com/paper/2106.02034/) (2021)
- [Focal Loss for Dense Object Detection](https://mlsanity.com/paper/1708.02002/) (2017)
- [Exploring Plain Vision Transformer Backbones for Object Detection](https://mlsanity.com/paper/2203.16527/) (2022)
- [DAB-DETR: Dynamic Anchor Boxes are Better Queries for DETR](https://mlsanity.com/paper/2201.12329/) (2022)
- [DN-DETR: Accelerate DETR Training by Introducing Query DeNoising](https://mlsanity.com/paper/2203.01305/) (2022)
- [Conditional DETR for Fast Training Convergence](https://mlsanity.com/paper/2108.06152/) (2021)
- [AdaViT: Adaptive Tokens for Efficient Vision Transformer](https://mlsanity.com/paper/2112.07658/) (2021)
- [Evo-ViT: Slow-Fast Token Evolution for Dynamic Vision Transformer](https://mlsanity.com/paper/2108.01390/) (2021)
- [Fast Convergence of DETR with Spatially Modulated Co-Attention](https://mlsanity.com/paper/2101.07448/) (2021)
- [You Only Look at One Sequence: Rethinking Transformer in Vision through Object Detection](https://mlsanity.com/paper/2106.00666/) (2021)
- [Rethinking Transformer-based Set Prediction for Object Detection](https://mlsanity.com/paper/2011.10881/) (2020)
- [Learned Token Pruning for Transformers](https://mlsanity.com/paper/2107.00910/) (2021)
- [Patch Slimming for Efficient Vision Transformers](https://mlsanity.com/paper/2106.02852/) (2021)
- [Efficient DETR: Improving End-to-End Object Detector with Dense Prior](https://mlsanity.com/paper/2104.01318/) (2021)
- [IA-RED2: Interpretability-Aware Redundancy Reduction for Vision Transformers](https://mlsanity.com/paper/2106.12620/) (2021)
- [Sparse DETR: Efficient End-to-End Object Detection with Learnable Sparsity](https://mlsanity.com/paper/2111.14330/) (2021)
- [Accelerating DETR Convergence via Semantic-Aligned Matching](https://mlsanity.com/paper/2203.06883/) (2022)
- [PnP-DETR: Towards Efficient Visual Analysis with Transformers](https://mlsanity.com/paper/2109.07036/) (2021)
- [Adaptive Sparse ViT: Towards Learnable Adaptive Token Pruning by Fully Exploiting Self-Attention](https://mlsanity.com/paper/2209.13802/) (2022)
- [Conditional DETR V2: Efficient Detection Transformer with Box Queries](https://mlsanity.com/paper/2207.08914/) (2022)
- [Towards Efficient Use of Multi-Scale Features in Transformer-Based Object Detectors](https://mlsanity.com/paper/2208.11356/) (2022)

## Built on by (most cited first, 30 of 38)

- [Gold-YOLO: Efficient Object Detector via Gather-and-Distribute Mechanism](https://mlsanity.com/paper/2309.11331/) (2023)
- [A Survey on Transformer Compression](https://mlsanity.com/paper/2402.05964/) (2024)
- [LW-DETR: A Transformer Replacement to YOLO for Real-Time Detection](https://mlsanity.com/paper/2406.03459/) (2024)
- [Ivy-Fake: A Unified Explainable Framework and Benchmark for Image and Video AIGC Detection](https://mlsanity.com/paper/2506.00979/) (2025)
- [Salience DETR: Enhancing Detection Transformer with Hierarchical Salience Filtering Refinement](https://mlsanity.com/paper/2403.16131/) (2024)
- [BAM-DETR: Boundary-Aligned Moment Detection Transformer for Temporal Sentence Grounding in Videos](https://mlsanity.com/paper/2312.00083/) (2023)
- [Towards Accurate Post-Training Quantization of Vision Transformers via Error Reduction](https://mlsanity.com/paper/2407.06794/) (2024)
- [Knowing Your Target: Target-Aware Transformer Makes Better Spatio-Temporal Video Grounding](https://mlsanity.com/paper/2502.11168/) (2025)
- [SLAB: Efficient Transformers with Simplified Linear Attention and Progressive Re-parameterized Batch Normalization](https://mlsanity.com/paper/2405.11582/) (2024)
- [LP-DETR: Layer-wise Progressive Relations for Object Detection](https://mlsanity.com/paper/2502.05147/) (2025)
- [Dynamic Dictionary Learning for Remote Sensing Image Segmentation](https://mlsanity.com/paper/2503.06683/) (2025)
- [TransGOP: Transformer-Based Gaze Object Prediction](https://mlsanity.com/paper/2402.13578/) (2024)
- [Rethinking Remote Sensing Change Detection With A Mask View](https://mlsanity.com/paper/2406.15320/) (2024)
- [Trexplorer Super: Topologically Correct Centerline Tree Tracking of Tubular Objects in CT Volumes](https://mlsanity.com/paper/2507.10881/) (2025)
- [Prompt-Aware Adapter: Towards Learning Adaptive Visual Tokens for Multimodal Large Language Models](https://mlsanity.com/paper/2405.15684/) (2024)
- [Token Pruning using a Lightweight Background Aware Vision Transformer](https://mlsanity.com/paper/2410.09324/) (2024)
- [FILA: Fine-Grained Vision Language Models](https://mlsanity.com/paper/2412.08378/) (2024)
- [Semantic Alignment and Reinforcement for Data-Free Quantization of Vision Transformers](https://mlsanity.com/paper/2412.16553/) (2024)
- [AHCQ-SAM: Toward Accurate and Hardware-Compatible Post-Training Segment Anything Model Quantization](https://mlsanity.com/paper/2503.03088/) (2025)
- [Conformal Object Detection by Sequential Risk Control](https://mlsanity.com/paper/2505.24038/) (2025)
- [DECO: Unleashing the Potential of ConvNets for Query-based Detection and Segmentation](https://mlsanity.com/paper/2312.13735/) (2023)
- [FastTextSpotter: A High-Efficiency Transformer for Multilingual Scene Text Spotting](https://mlsanity.com/paper/2408.14998/) (2024)
- [DI-MaskDINO: A Joint Object Detection and Instance Segmentation Model](https://mlsanity.com/paper/2410.16707/) (2024)
- [YOLO-TLA: An Efficient and Lightweight Small Object Detection Model based on YOLOv5](https://mlsanity.com/paper/2402.14309/) (2024)
- [Cross-DINO: Cross the Deep MLP and Transformer for Small Object Detection](https://mlsanity.com/paper/2505.21868/) (2025)
- [DS-Det: Single-Query Paradigm and Attention Disentangled Learning for Flexible Object Detection](https://mlsanity.com/paper/2507.19807/) (2025)
- [Index-Preserving Lightweight Token Pruning for Efficient Document Understanding in Vision-Language Models](https://mlsanity.com/paper/2509.06415/) (2025)
- [SpineBench: A Clinically Salient, Level-Aware Benchmark Powered by the SpineMed-450k Corpus](https://mlsanity.com/paper/2510.03160/) (2025)
- [MOBIUS: Big-to-Mobile Universal Instance Segmentation via Multi-modal Bottleneck Fusion and Calibrated Decoder Pruning](https://mlsanity.com/paper/2510.15026/) (2025)
- [Towards Lossless Ultimate Vision Token Compression for VLMs](https://mlsanity.com/paper/2512.09010/) (2025)

## Abstract (arXiv)

DETR-like models have significantly boosted the performance of detectors and even outperformed classical convolutional models. However, all tokens are treated equally without discrimination brings a redundant computational burden in the traditional encoder structure. The recent sparsification strategies exploit a subset of informative tokens to reduce attention complexity maintaining performance through the sparse encoder. But these methods tend to rely on unreliable model statistics. Moreover, simply reducing the token population hinders the detection performance to a large extent, limiting the application of these sparse models. We propose Focus-DETR, which focuses attention on more informative tokens for a better trade-off between computation efficiency and model accuracy. Specifically, we reconstruct the encoder with dual attention, which includes a token scoring mechanism that considers both localization and category semantic information of the objects from multi-scale feature maps. We efficiently abandon the background queries and enhance the semantic interaction of the fine-grained object queries based on the scores. Compared with the state-of-the-art sparse DETR-like detectors under the same setting, our Focus-DETR gets comparable complexity while achieving 50.4AP (+2.2) on COCO. The code is available at https://github.com/huawei-noah/noah-research/tree/master/Focus-DETR and https://gitee.com/mindspore/models/tree/master/research/cv/Focus-DETR.