# Breaking Self-Attention Failure: Rethinking Query Initialization for Infrared Small Target Detection

arXiv: https://arxiv.org/abs/2601.02837 · HTML: https://mlsanity.com/paper/2601.02837/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 9 of its 18 references that are in the map.
- First posted on arXiv 6 Jan 2026.

## Builds on (most cited first, 9 of 9)

- [Deformable DETR: Deformable Transformers for End-to-End Object Detection](https://mlsanity.com/paper/2010.04159/) (2020)
- [DINO: DETR with Improved DeNoising Anchor Boxes for End-to-End Object Detection](https://mlsanity.com/paper/2203.03605/) (2022)
- [DN-DETR: Accelerate DETR Training by Introducing Query DeNoising](https://mlsanity.com/paper/2203.01305/) (2022)
- [Infrared Small Target Detection with Scale and Location Sensitivity](https://mlsanity.com/paper/2403.19366/) (2024)
- [C2Former: Calibrated and Complementary Transformer for RGB-Infrared Object Detection](https://mlsanity.com/paper/2306.16175/) (2023)
- [Translation, Scale and Rotation: Cross-Modal Alignment Meets RGB-Infrared Vehicle Detection](https://mlsanity.com/paper/2209.13801/) (2022)
- [UniRGB-IR: A Unified Framework for Visible-Infrared Semantic Tasks via Adapter Tuning](https://mlsanity.com/paper/2404.17360/) (2024)
- [AGPCNet: Attention-Guided Pyramid Context Networks for Infrared Small Target Detection](https://mlsanity.com/paper/2111.03580/) (2021)
- [Rethinking Multi-Modal Object Detection from the Perspective of Mono-Modality Feature Learning](https://mlsanity.com/paper/2503.11780/) (2025)

## Abstract (arXiv)

Infrared small target detection (IRSTD) faces significant challenges due to low signal-to-noise ratios, extremely small target sizes, and complex cluttered backgrounds. Although DETR-based detectors benefit from global context modeling, their query initialization can become unreliable in IRSTD because only a few encoder tokens correspond to targets, while the majority describe the background. We revisit this phenomenon and reveal that the target-relevant embeddings of IRST are inevitably overwhelmed by dominant background features due to the self-attention mechanism, leading to unreliable query initialization and inaccurate target localization. To address this issue, we propose SEF-DETR, a novel framework that refines query initialization through Patch-wise Spectral Screening (PSS), Frequency-Routed Examination (FRE), and Reliability-Consistency-aware Fusion (RCF). PSS encodes the radial and directional energy distributions of local Fourier spectra and aggregates overlapping patch predictions into a target-relevant density map. Guided by this map, FRE performs sparse deformable re-examination after each encoder layer, while retaining content-driven sampling to distinguish true targets from frequency-domain false alarms. RCF further re-ranks candidate queries according to spatial-frequency consistency and frequency reliability. Extensive experiments on three public IRSTD datasets demonstrate that SEF-DETR achieves superior detection performance over state-of-the-art methods with low computational overhead, providing a robust and efficient DETR-based solution for IRSTD.