# Lite DETR : An Interleaved Multi-Scale Encoder for Efficient DETR

arXiv: https://arxiv.org/abs/2303.07335 · HTML: https://mlsanity.com/paper/2303.07335/

## Key facts

- Cited by 36 papers in the map (data through 14 Oct 2026).
- Builds on 32 of its 38 references that are in the map.
- First posted on arXiv 13 Mar 2023.

## Builds on (most cited first, 30 of 32)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Feature Pyramid Networks for Object Detection](https://mlsanity.com/paper/1612.03144/) (2016)
- [Deformable DETR: Deformable Transformers for End-to-End Object Detection](https://mlsanity.com/paper/2010.04159/) (2020)
- [YOLOv3: An Incremental Improvement](https://mlsanity.com/paper/1804.02767/) (2018)
- [Masked-attention Mask Transformer for Universal Image Segmentation](https://mlsanity.com/paper/2112.01527/) (2021)
- [DINO: DETR with Improved DeNoising Anchor Boxes for End-to-End Object Detection](https://mlsanity.com/paper/2203.03605/) (2022)
- [Per-Pixel Classification is Not All You Need for Semantic Segmentation](https://mlsanity.com/paper/2107.06278/) (2021)
- [EfficientDet: Scalable and Efficient Object Detection](https://mlsanity.com/paper/1911.09070/) (2019)
- [YOLOv7: Trainable bag-of-freebies sets new state-of-the-art for real-time object detectors](https://mlsanity.com/paper/2207.02696/) (2022)
- [Path Aggregation Network for Instance Segmentation](https://mlsanity.com/paper/1803.01534/) (2018)
- [DAB-DETR: Dynamic Anchor Boxes are Better Queries for DETR](https://mlsanity.com/paper/2201.12329/) (2022)
- [DN-DETR: Accelerate DETR Training by Introducing Query DeNoising](https://mlsanity.com/paper/2203.01305/) (2022)
- [Hybrid Task Cascade for Instance Segmentation](https://mlsanity.com/paper/1901.07518/) (2019)
- [Conditional DETR for Fast Training Convergence](https://mlsanity.com/paper/2108.06152/) (2021)
- [Mask DINO: Towards A Unified Transformer-based Framework for Object Detection and Segmentation](https://mlsanity.com/paper/2206.02777/) (2022)
- [NAS-FPN: Learning Scalable Feature Pyramid Architecture for Object Detection](https://mlsanity.com/paper/1904.07392/) (2019)
- [DetectoRS: Detecting Objects with Recursive Feature Pyramid and Switchable Atrous Convolution](https://mlsanity.com/paper/2006.02334/) (2020)
- [DETRs with Hybrid Matching](https://mlsanity.com/paper/2207.13080/) (2022)
- [Efficient DETR: Improving End-to-End Object Detector with Dense Prior](https://mlsanity.com/paper/2104.01318/) (2021)
- [Sparse DETR: Efficient End-to-End Object Detection with Learnable Sparsity](https://mlsanity.com/paper/2111.14330/) (2021)
- [AdaMixer: A Fast-Converging Query-Based Object Detector](https://mlsanity.com/paper/2203.16507/) (2022)
- [PnP-DETR: Towards Efficient Visual Analysis with Transformers](https://mlsanity.com/paper/2109.07036/) (2021)
- [End-to-End Trainable Multi-Instance Pose Estimation with Transformers](https://mlsanity.com/paper/2103.12115/) (2021)
- [D^2ETR: Decoder-Only DETR with Computationally Efficient Cross-Scale Attention](https://mlsanity.com/paper/2203.00860/) (2022)
- [Towards Efficient Use of Multi-Scale Features in Transformer-Based Object Detectors](https://mlsanity.com/paper/2208.11356/) (2022)
- [An Extendable, Efficient and Effective Transformer-based Object Detector](https://mlsanity.com/paper/2204.07962/) (2022)

## Built on by (most cited first, 30 of 36)

- [DETRs Beat YOLOs on Real-time Object Detection](https://mlsanity.com/paper/2304.08069/) (2023)
- [Transformer-Based Visual Segmentation: A Survey](https://mlsanity.com/paper/2304.09854/) (2023)
- [Grounding DINO 1.5: Advance the "Edge" of Open-Set Object Detection](https://mlsanity.com/paper/2405.10300/) (2024)
- [DEIM: DETR with Improved Matching for Fast Convergence](https://mlsanity.com/paper/2412.04234/) (2024)
- [TAPTR: Tracking Any Point with Transformers as Detection](https://mlsanity.com/paper/2403.13042/) (2024)
- [Vote2Cap-DETR++: Decoupling Localization and Describing for End-to-End 3D Dense Captioning](https://mlsanity.com/paper/2309.02999/) (2023)
- [LW-DETR: A Transformer Replacement to YOLO for Real-Time Detection](https://mlsanity.com/paper/2406.03459/) (2024)
- [Salience DETR: Enhancing Detection Transformer with Hierarchical Salience Filtering Refinement](https://mlsanity.com/paper/2403.16131/) (2024)
- [BAM-DETR: Boundary-Aligned Moment Detection Transformer for Temporal Sentence Grounding in Videos](https://mlsanity.com/paper/2312.00083/) (2023)
- [TAPTRv2: Attention-based Position Update Improves Tracking Any Point](https://mlsanity.com/paper/2407.16291/) (2024)
- [Hausdorff Distance Matching with Adaptive Query Denoising for Rotated Detection Transformer](https://mlsanity.com/paper/2305.07598/) (2023)
- [TAPTRv3: Spatial and Temporal Context Foster Robust Tracking of Any Point in Long Video](https://mlsanity.com/paper/2411.18671/) (2024)
- [Align-DETR: Enhancing End-to-end Object Detection with Aligned Loss](https://mlsanity.com/paper/2304.07527/) (2023)
- [LeanGaussian: Breaking Pixel or Point Cloud Correspondence in Modeling 3D Gaussians](https://mlsanity.com/paper/2404.16323/) (2024)
- [CASPFormer: Trajectory Prediction from BEV Images with Deformable Attention](https://mlsanity.com/paper/2409.17790/) (2024)
- [Self-Supervised Visual Prompting for Cross-Domain Road Damage Detection](https://mlsanity.com/paper/2511.12410/) (2025)
- [Box-DETR: Understanding and Boxing Conditional Spatial Queries](https://mlsanity.com/paper/2307.08353/) (2023)
- [DECO: Unleashing the Potential of ConvNets for Query-based Detection and Segmentation](https://mlsanity.com/paper/2312.13735/) (2023)
- [DI-MaskDINO: A Joint Object Detection and Instance Segmentation Model](https://mlsanity.com/paper/2410.16707/) (2024)
- [Referencing Where to Focus: Improving VisualGrounding with Referential Query](https://mlsanity.com/paper/2412.19155/) (2024)
- [Don't let the information slip away](https://mlsanity.com/paper/2602.22595/) (2026)
- [Progressive Token Length Scaling in Transformer Encoders for Efficient Universal Segmentation](https://mlsanity.com/paper/2404.14657/) (2024)
- [Synchronized Object Detection for Autonomous Sorting, Mapping, and Quantification of Materials in Circular Healthcare](https://mlsanity.com/paper/2405.06821/) (2024)
- [Global Clipper: Enhancing Safety and Reliability of Transformer-based Object Detection Models](https://mlsanity.com/paper/2406.03229/) (2024)
- [Mr. DETR++: Instructive Multi-Route Training for Detection Transformers with Mixture-of-Experts](https://mlsanity.com/paper/2412.10028/) (2024)
- [MOBIUS: Big-to-Mobile Universal Instance Segmentation via Multi-modal Bottleneck Fusion and Calibrated Decoder Pruning](https://mlsanity.com/paper/2510.15026/) (2025)
- [CP-WSP: A Declarative CP-SAT Framework for Configurable Multi-Constraint Workforce Scheduling](https://mlsanity.com/paper/2607.05177/) (2026)
- [Efficient Pre-training for Localized Instruction Generation of Videos](https://mlsanity.com/paper/2311.15964/) (2023)
- [Image-Specific Adaptation of Transformer Encoders for Compute-Efficient Segmentation](https://mlsanity.com/paper/2404.15244/) (2024)
- [Cross Resolution Encoding-Decoding For Detection Transformers](https://mlsanity.com/paper/2410.04088/) (2024)

## Abstract (arXiv)

Recent DEtection TRansformer-based (DETR) models have obtained remarkable performance. Its success cannot be achieved without the re-introduction of multi-scale feature fusion in the encoder. However, the excessively increased tokens in multi-scale features, especially for about 75% of low-level features, are quite computationally inefficient, which hinders real applications of DETR models. In this paper, we present Lite DETR, a simple yet efficient end-to-end object detection framework that can effectively reduce the GFLOPs of the detection head by 60% while keeping 99% of the original performance. Specifically, we design an efficient encoder block to update high-level features (corresponding to small-resolution feature maps) and low-level features (corresponding to large-resolution feature maps) in an interleaved way. In addition, to better fuse cross-scale features, we develop a key-aware deformable attention to predict more reliable attention weights. Comprehensive experiments validate the effectiveness and efficiency of the proposed Lite DETR, and the efficient encoder strategy can generalize well across existing DETR-based models. The code will be available in urlhttps://github.com/IDEA-Research/Lite-DETR.