# LW-DETR: A Transformer Replacement to YOLO for Real-Time Detection

arXiv: https://arxiv.org/abs/2406.03459 · HTML: https://mlsanity.com/paper/2406.03459/

## Key facts

- Cited by 36 papers in the map (data through 14 Oct 2026).
- Builds on 60 of its 73 references that are in the map.
- First posted on arXiv 5 Jun 2024.

## Builds on (most cited first, 30 of 60)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016)
- [mixup: Beyond Empirical Risk Minimization](https://mlsanity.com/paper/1710.09412/) (2017)
- [You Only Look Once: Unified, Real-Time Object Detection](https://mlsanity.com/paper/1506.02640/) (2015)
- [Deformable DETR: Deformable Transformers for End-to-End Object Detection](https://mlsanity.com/paper/2010.04159/) (2020)
- [Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutions](https://mlsanity.com/paper/2102.12122/) (2021)
- [YOLOv4: Optimal Speed and Accuracy of Object Detection](https://mlsanity.com/paper/2004.10934/) (2020)
- [FCOS: Fully Convolutional One-Stage Object Detection](https://mlsanity.com/paper/1904.01355/) (2019)
- [Generalized Intersection over Union: A Metric and A Loss for Bounding Box Regression](https://mlsanity.com/paper/1902.09630/) (2019)
- [DINO: DETR with Improved DeNoising Anchor Boxes for End-to-End Object Detection](https://mlsanity.com/paper/2203.03605/) (2022)
- [Planning-oriented Autonomous Driving](https://mlsanity.com/paper/2212.10156/) (2022)
- [YOLOX: Exceeding YOLO Series in 2021](https://mlsanity.com/paper/2107.08430/) (2021)
- [Scaling Vision Transformers](https://mlsanity.com/paper/2106.04560/) (2021)
- [YOLOv7: Trainable bag-of-freebies sets new state-of-the-art for real-time object detectors](https://mlsanity.com/paper/2207.02696/) (2022)
- [Exploring Plain Vision Transformer Backbones for Object Detection](https://mlsanity.com/paper/2203.16527/) (2022)
- [DAB-DETR: Dynamic Anchor Boxes are Better Queries for DETR](https://mlsanity.com/paper/2201.12329/) (2022)
- [DETRs Beat YOLOs on Real-time Object Detection](https://mlsanity.com/paper/2304.08069/) (2023)
- [RepVGG: Making VGG-style ConvNets Great Again](https://mlsanity.com/paper/2101.03697/) (2021)
- [YOLOv10: Real-Time End-to-End Object Detection](https://mlsanity.com/paper/2405.14458/) (2024)
- [DN-DETR: Accelerate DETR Training by Introducing Query DeNoising](https://mlsanity.com/paper/2203.01305/) (2022)
- [Distance-IoU Loss: Faster and Better Learning for Bounding Box Regression](https://mlsanity.com/paper/1911.08287/) (2019)
- [Conditional DETR for Fast Training Convergence](https://mlsanity.com/paper/2108.06152/) (2021)
- [CSPNet: A New Backbone that can Enhance Learning Capability of CNN](https://mlsanity.com/paper/1911.11929/) (2019)
- [Deep Multi-modal Object Detection and Semantic Segmentation for Autonomous Driving: Datasets, Methods, and Challenges](https://mlsanity.com/paper/1902.07830/) (2019)

## Built on by (most cited first, 30 of 36)

- [DINO-X: A Unified Vision Model for Open-World Object Detection and Understanding](https://mlsanity.com/paper/2411.14347/) (2024)
- [RF-DETR: Neural Architecture Search for Real-Time Detection Transformers](https://mlsanity.com/paper/2511.09554/) (2025)
- [D-FINE: Redefine Regression Task in DETRs as Fine-grained Distribution Refinement](https://mlsanity.com/paper/2410.13842/) (2024)
- [Roboflow100-VL: A Multi-Domain Object Detection Benchmark for Vision-Language Models](https://mlsanity.com/paper/2505.20612/) (2025)
- [LaMI-DETR: Open-Vocabulary Detection with Language Model Instruction](https://mlsanity.com/paper/2407.11335/) (2024)
- [Dome-DETR: DETR with Density-Oriented Feature-Query Manipulation for Efficient Tiny Object Detection](https://mlsanity.com/paper/2505.05741/) (2025)
- [FastPose-ViT: A Vision Transformer for Real-Time Spacecraft Pose Estimation](https://mlsanity.com/paper/2512.09792/) (2025)
- [OVLW-DETR: Open-Vocabulary Light-Weighted Detection Transformer](https://mlsanity.com/paper/2407.10655/) (2024)
- [Create Anything Anywhere: Layout-Controllable Personalized Diffusion Model for Multiple Subjects](https://mlsanity.com/paper/2505.20909/) (2025)
- [EdgeCrafter: Compact ViTs for Edge Dense Prediction via Task-Specialized Distillation](https://mlsanity.com/paper/2603.18739/) (2026)
- [DMAT: An End-to-End Framework for Joint Atmospheric Turbulence Mitigation and Object Detection](https://mlsanity.com/paper/2507.04323/) (2025)
- [PK-YOLO: Pretrained Knowledge Guided YOLO for Brain Tumor Detection in Multiplanar MRI Slices](https://mlsanity.com/paper/2410.21822/) (2024)
- [Mr. DETR++: Instructive Multi-Route Training for Detection Transformers with Mixture-of-Experts](https://mlsanity.com/paper/2412.10028/) (2024)
- [DETRPose: Real-Time End-to-End Multi-Person Pose Estimation via Modified Transformer Decoder and Novel Denoising Keypoints](https://mlsanity.com/paper/2506.13027/) (2025)
- [JDATT: A Joint Distillation Framework for Atmospheric Turbulence Mitigation and Target Detection](https://mlsanity.com/paper/2507.19780/) (2025)
- [FruitProm: Probabilistic Maturity Estimation and Detection of Fruits and Vegetables](https://mlsanity.com/paper/2510.24885/) (2025)
- [LeAD-M3D: Leveraging Asymmetric Distillation for Real-Time Monocular 3D Detection](https://mlsanity.com/paper/2512.05663/) (2025)
- [RiO-DETR: DETR for Real-time Oriented Object Detection](https://mlsanity.com/paper/2603.09411/) (2026)
- [WUTDet: A 100K-Scale Ship Detection Dataset and Benchmarks with Dense Small Objects](https://mlsanity.com/paper/2604.07759/) (2026)
- [RT-SDGOD: Real-Time Single-Domain Generalized Object Detection](https://mlsanity.com/paper/2606.09367/) (2026)
- [HOIBlender: Blending Lightweight Detection with Vision-Language Priors for Efficient Human-Object Interaction Detection](https://mlsanity.com/paper/2609.23431/) (2026)
- [A Distributed Framework for Privacy-Enhanced Vision Transformers on the Edge](https://mlsanity.com/paper/2512.09309/) (2025)
- [Comparing Object Detection Models for Electrical Substation Component Mapping](https://mlsanity.com/paper/2512.22454/) (2025)
- [FlyPose: Towards Robust Human Pose Estimation From Aerial Views](https://mlsanity.com/paper/2601.05747/) (2026)
- [Multimodal Urban Tree Detection from Satellite and Street-Level Imagery via Annotation-Efficient Deep Learning Strategies](https://mlsanity.com/paper/2604.03505/) (2026)
- [SynSur: An end-to-end generative pipeline for synthetic industrial surface defect generation and detection](https://mlsanity.com/paper/2604.26633/) (2026)
- [Echo-α: Large Agentic Multimodal Reasoning Model for Ultrasound Interpretation](https://mlsanity.com/paper/2604.28011/) (2026)
- [Real-Time Source-Free Object Detection](https://mlsanity.com/paper/2606.31834/) (2026)
- [Enabling Fully Integer-Only Inference for Lightweight Detection Transformers](https://mlsanity.com/paper/2607.24981/) (2026)
- [ScratchSim: A Procedural Synthetic Data Pipeline for Surface Scratch Detection](https://mlsanity.com/paper/2607.27065/) (2026)

## Abstract (arXiv)

In this paper, we present a light-weight detection transformer, LW-DETR, which outperforms YOLOs for real-time object detection. The architecture is a simple stack of a ViT encoder, a projector, and a shallow DETR decoder. Our approach leverages recent advanced techniques, such as training-effective techniques, e.g., improved loss and pretraining, and interleaved window and global attentions for reducing the ViT encoder complexity. We improve the ViT encoder by aggregating multi-level feature maps, and the intermediate and final feature maps in the ViT encoder, forming richer feature maps, and introduce window-major feature map organization for improving the efficiency of interleaved attention computation. Experimental results demonstrate that the proposed approach is superior over existing real-time detectors, e.g., YOLO and its variants, on COCO and other benchmark datasets. Code and models are available at (https://github.com/Atten4Vis/LW-DETR).