# Salience DETR: Enhancing Detection Transformer with Hierarchical Salience Filtering Refinement

arXiv: https://arxiv.org/abs/2403.16131 · HTML: https://mlsanity.com/paper/2403.16131/

## Key facts

- Cited by 31 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 37 references that are in the map.
- First posted on arXiv 24 Mar 2024.

## Builds on (most cited first, 29 of 29)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- [You Only Look Once: Unified, Real-Time Object Detection](https://mlsanity.com/paper/1506.02640/) (2015)
- [Deformable DETR: Deformable Transformers for End-to-End Object Detection](https://mlsanity.com/paper/2010.04159/) (2020)
- [SSD: Single Shot MultiBox Detector](https://mlsanity.com/paper/1512.02325/) (2015)
- [MMDetection: Open MMLab Detection Toolbox and Benchmark](https://mlsanity.com/paper/1906.07155/) (2019)
- [DINO: DETR with Improved DeNoising Anchor Boxes for End-to-End Object Detection](https://mlsanity.com/paper/2203.03605/) (2022)
- [Cascade R-CNN: Delving into High Quality Object Detection](https://mlsanity.com/paper/1712.00726/) (2017)
- [YOLOX: Exceeding YOLO Series in 2021](https://mlsanity.com/paper/2107.08430/) (2021)
- [Path Aggregation Network for Instance Segmentation](https://mlsanity.com/paper/1803.01534/) (2018)
- [DAB-DETR: Dynamic Anchor Boxes are Better Queries for DETR](https://mlsanity.com/paper/2201.12329/) (2022)
- [DN-DETR: Accelerate DETR Training by Introducing Query DeNoising](https://mlsanity.com/paper/2203.01305/) (2022)
- [Hybrid Task Cascade for Instance Segmentation](https://mlsanity.com/paper/1901.07518/) (2019)
- [Conditional DETR for Fast Training Convergence](https://mlsanity.com/paper/2108.06152/) (2021)
- [DETRs with Collaborative Hybrid Assignments Training](https://mlsanity.com/paper/2211.12860/) (2022)
- [TOOD: Task-aligned One-stage Object Detection](https://mlsanity.com/paper/2108.07755/) (2021)
- [DETRs with Hybrid Matching](https://mlsanity.com/paper/2207.13080/) (2022)
- [Efficient DETR: Improving End-to-End Object Detector with Dense Prior](https://mlsanity.com/paper/2104.01318/) (2021)
- [AutoAssign: Differentiable Label Assignment for Dense Object Detection](https://mlsanity.com/paper/2007.03496/) (2020)
- [Sparse DETR: Efficient End-to-End Object Detection with Learnable Sparsity](https://mlsanity.com/paper/2111.14330/) (2021)
- [Group DETR: Fast DETR Training with Group-Wise One-to-Many Assignment](https://mlsanity.com/paper/2207.13085/) (2022)
- [Accelerating DETR Convergence via Semantic-Aligned Matching](https://mlsanity.com/paper/2203.06883/) (2022)
- [PnP-DETR: Towards Efficient Visual Analysis with Transformers](https://mlsanity.com/paper/2109.07036/) (2021)
- [Less is More: Focus Attention for Efficient DETR](https://mlsanity.com/paper/2307.12612/) (2023)
- [Lite DETR : An Interleaved Multi-Scale Encoder for Efficient DETR](https://mlsanity.com/paper/2303.07335/) (2023)
- [Cascade-DETR: Delving into High-Quality Universal Object Detection](https://mlsanity.com/paper/2307.11035/) (2023)

## Built on by (most cited first, 30 of 31)

- [FG-CLIP: Fine-Grained Visual and Textual Alignment](https://mlsanity.com/paper/2505.05071/) (2025)
- [Relation DETR: Exploring Explicit Position Relation Prior for Object Detection](https://mlsanity.com/paper/2407.11699/) (2024)
- [Psi-Sampler: Initial Particle Sampling for SMC-Based Inference-Time Reward Alignment in Score Models](https://mlsanity.com/paper/2506.01320/) (2025)
- [ORIGEN: Zero-Shot 3D Orientation Grounding in Text-to-Image Generation](https://mlsanity.com/paper/2503.22194/) (2025)
- [LMM-Det: Make Large Multimodal Models Excel in Object Detection](https://mlsanity.com/paper/2507.18300/) (2025)
- [MI-DETR: An Object Detection Model with Multi-time Inquiries Mechanism](https://mlsanity.com/paper/2503.01463/) (2025)
- [LP-DETR: Layer-wise Progressive Relations for Object Detection](https://mlsanity.com/paper/2502.05147/) (2025)
- [Referring Expression Comprehension for Small Objects](https://mlsanity.com/paper/2510.03701/) (2025)
- [PaQ-DETR: Learning Pattern and Quality-Aware Dynamic Queries for Object Detection](https://mlsanity.com/paper/2603.06917/) (2026)
- [Dual-R-DETR: Resolving Query Competition with Pairwise Routing in Transformer Decoders](https://mlsanity.com/paper/2512.13876/) (2025)
- [Integrating Diverse Assignment Strategies into DETRs](https://mlsanity.com/paper/2601.09247/) (2026)
- [PK-YOLO: Pretrained Knowledge Guided YOLO for Brain Tumor Detection in Multiplanar MRI Slices](https://mlsanity.com/paper/2410.21822/) (2024)
- [Mr. DETR++: Instructive Multi-Route Training for Detection Transformers with Mixture-of-Experts](https://mlsanity.com/paper/2412.10028/) (2024)
- [IV-tuning: Parameter-Efficient Transfer Learning for Infrared-Visible Tasks](https://mlsanity.com/paper/2412.16654/) (2024)
- [Fractional Correspondence Framework in Detection Transformer](https://mlsanity.com/paper/2503.04107/) (2025)
- [FruitProm: Probabilistic Maturity Estimation and Detection of Fruits and Vegetables](https://mlsanity.com/paper/2510.24885/) (2025)
- [Salience-SGG: Enhancing Unbiased Scene Graph Generation with Iterative Salience Estimation](https://mlsanity.com/paper/2601.08728/) (2026)
- [Caries DETR: Tooth Structure-aware Prior and Lesion-aware Dynamic Loss Refinement for DETR Based Caries Detection](https://mlsanity.com/paper/2604.23718/) (2026)
- [MDS-DETR: DETR with Masked Duplicate Suppressor](https://mlsanity.com/paper/2605.23507/) (2026)
- [Dynamic Universal Approximation Theory: The Basic Theory for Deep Learning-Based Computer Vision Models](https://mlsanity.com/paper/2407.17480/) (2024)
- [Cross Resolution Encoding-Decoding For Detection Transformers](https://mlsanity.com/paper/2410.04088/) (2024)
- [Fine-Grained Open-Vocabulary Object Recognition via User-Guided Segmentation](https://mlsanity.com/paper/2411.15620/) (2024)
- [BiXFormer: A Robust Framework for Maximizing Modality Effectiveness in Multi-Modal Semantic Segmentation](https://mlsanity.com/paper/2506.03675/) (2025)
- [PerioDet: Large-Scale Panoramic Radiograph Benchmark for Clinical-Oriented Apical Periodontitis Detection](https://mlsanity.com/paper/2507.18958/) (2025)
- [Revisiting DETR for Small Object Detection via Noise-Resilient Query Optimization](https://mlsanity.com/paper/2507.19059/) (2025)
- [A Unified Detection Pipeline for Robust Object Detection in Fisheye-Based Traffic Surveillance](https://mlsanity.com/paper/2510.20016/) (2025)
- [Prior-Guided DETR for Ultrasound Nodule Detection](https://mlsanity.com/paper/2601.02212/) (2026)
- [SToRe3D: Sparse Token Relevance in ViTs for Efficient Multi-View 3D Object Detection](https://mlsanity.com/paper/2605.14110/) (2026)
- [Parallel Tempering Initial Sampling in Inference-Time Reward Alignment](https://mlsanity.com/paper/2605.30991/) (2026)
- [When Fish Look Alike: Tracking Identities with Dual-branch Elasticity](https://mlsanity.com/paper/2607.26412/) (2026)

## Abstract (arXiv)

DETR-like methods have significantly increased detection performance in an end-to-end manner. The mainstream two-stage frameworks of them perform dense self-attention and select a fraction of queries for sparse cross-attention, which is proven effective for improving performance but also introduces a heavy computational burden and high dependence on stable query selection. This paper demonstrates that suboptimal two-stage selection strategies result in scale bias and redundancy due to the mismatch between selected queries and objects in two-stage initialization. To address these issues, we propose hierarchical salience filtering refinement, which performs transformer encoding only on filtered discriminative queries, for a better trade-off between computational efficiency and precision. The filtering process overcomes scale bias through a novel scale-independent salience supervision. To compensate for the semantic misalignment among queries, we introduce elaborate query refinement modules for stable two-stage initialization. Based on above improvements, the proposed Salience DETR achieves significant improvements of +4.0% AP, +0.2% AP, +4.4% AP on three challenging task-specific detection datasets, as well as 49.2% AP on COCO 2017 with less FLOPs. The code is available at https://github.com/xiuqhou/Salience-DETR.