# RT-DETRv4: Painlessly Furthering Real-Time Object Detection with Vision Foundation Models

arXiv: https://arxiv.org/abs/2510.25257 · HTML: https://mlsanity.com/paper/2510.25257/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 29 Oct 2025.

## Built on by (most cited first, 15 of 15)

- [Ultralytics YOLO26: Unified Real-Time End-to-End Vision Models](https://mlsanity.com/paper/2606.03748/) (2026)
- [EdgeCrafter: Compact ViTs for Edge Dense Prediction via Task-Specialized Distillation](https://mlsanity.com/paper/2603.18739/) (2026)
- [A Comparative Evaluation of Large Vision-Language Models for 2D Object Detection under SOTIF Conditions](https://mlsanity.com/paper/2601.22830/) (2026)
- [RiO-DETR: DETR for Real-time Oriented Object Detection](https://mlsanity.com/paper/2603.09411/) (2026)
- [RT-SDGOD: Real-Time Single-Domain Generalized Object Detection](https://mlsanity.com/paper/2606.09367/) (2026)
- [CP-WSP: A Declarative CP-SAT Framework for Configurable Multi-Constraint Workforce Scheduling](https://mlsanity.com/paper/2607.05177/) (2026)
- [Detection of On-Ground Chestnuts Using Artificial Intelligence Toward Automated Picking](https://mlsanity.com/paper/2602.14140/) (2026)
- [DINOv3 Meets YOLO26 for Weed Detection in Vegetable Crops](https://mlsanity.com/paper/2603.00160/) (2026)
- [VFM4SDG: Unveiling the Power of VFMs for Single-Domain Generalized Object Detection](https://mlsanity.com/paper/2604.21502/) (2026)
- [Training-Free Tunnel Defect Inspection and Engineering Interpretation via Visual Recalibration and Entity Reconstruction](https://mlsanity.com/paper/2604.27928/) (2026)
- [GTR: Gated Token Recurrence for Efficient Dense Prediction](https://mlsanity.com/paper/2609.26590/) (2026)
- [GAD-MambaUNet: Direction-Group Mamba with Gradient-Adaptive DINOv3 Distillation for Lightweight Medical Image Segmentation](https://mlsanity.com/paper/2609.26729/) (2026)
- [MicroQonv: Reshaping Convolution Tensors for Efficient Microscaling in Training and Inference](https://mlsanity.com/paper/2609.28358/) (2026)
- [DeskForge: Dense Supervision from Desktop Environments for Computer-Use Agents](https://mlsanity.com/paper/2610.02320/) (2026)
- [RYOPO: Bringing End-to-End Category-Level Object Pose Estimation into Real Time](https://mlsanity.com/paper/2610.03013/) (2026)

## Abstract (arXiv)

Real-time object detection has achieved substantial progress through meticulously designed architectures and optimization strategies. However, the pursuit of high-speed inference via lightweight network designs often leads to degraded feature representation, which hinders further performance improvements and practical on-device deployment. In this paper, we propose a cost-effective and highly adaptable distillation framework that harnesses the rapidly evolving capabilities of Vision Foundation Models (VFMs) to enhance lightweight object detectors. Given the significant architectural and learning objective disparities between VFMs and resource-constrained detectors, achieving stable and task-aligned semantic transfer is challenging. To address this, on one hand, we introduce a Deep Semantic Injector (DSI) module that facilitates the integration of high-level representations from VFMs into the deep layers of the detector. On the other hand, we devise a Gradient-guided Adaptive Modulation (GAM) strategy, which dynamically adjusts the intensity of semantic transfer based on gradient norm ratios. Without increasing deployment and inference overhead, our approach painlessly delivers striking and consistent performance gains across diverse DETR-based models, underscoring its practical utility for real-time detection. Our new model family, RT-DETRv4, achieves state-of-the-art results on COCO, attaining AP scores of 49.8/53.7/55.4/57.0 at corresponding speeds of 273/169/124/78 FPS. Code is publicly available at https://github.com/RT-DETRs/RT-DETRv4.