# RF-DETR: Neural Architecture Search for Real-Time Detection Transformers

arXiv: https://arxiv.org/abs/2511.09554 · HTML: https://mlsanity.com/paper/2511.09554/

## Key facts

- Cited by 75 papers in the map (data through 14 Oct 2026).
- Builds on 45 of its 54 references that are in the map.
- First posted on arXiv 12 Nov 2025.

## Builds on (most cited first, 30 of 45)

- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- [You Only Look Once: Unified, Real-Time Object Detection](https://mlsanity.com/paper/1506.02640/) (2015)
- [Deformable DETR: Deformable Transformers for End-to-End Object Detection](https://mlsanity.com/paper/2010.04159/) (2020)
- [Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection](https://mlsanity.com/paper/2303.05499/) (2023)
- [SSD: Single Shot MultiBox Detector](https://mlsanity.com/paper/1512.02325/) (2015)
- [SAM 2: Segment Anything in Images and Videos](https://mlsanity.com/paper/2408.00714/) (2024)
- [Neural Architecture Search with Reinforcement Learning](https://mlsanity.com/paper/1611.01578/) (2016)
- [DINOv3](https://mlsanity.com/paper/2508.10104/) (2025)
- [DINO: DETR with Improved DeNoising Anchor Boxes for End-to-End Object Detection](https://mlsanity.com/paper/2203.03605/) (2022)
- [Grounded Language-Image Pre-training](https://mlsanity.com/paper/2112.03857/) (2021)
- [Learning Transferable Architectures for Scalable Image Recognition](https://mlsanity.com/paper/1707.07012/) (2017)
- [EfficientDet: Scalable and Efficient Object Detection](https://mlsanity.com/paper/1911.09070/) (2019)
- [YOLOv7: Trainable bag-of-freebies sets new state-of-the-art for real-time object detectors](https://mlsanity.com/paper/2207.02696/) (2022)
- [MnasNet: Platform-Aware Neural Architecture Search for Mobile](https://mlsanity.com/paper/1807.11626/) (2018)
- [Regularized Evolution for Image Classifier Architecture Search](https://mlsanity.com/paper/1802.01548/) (2018)
- [Once-for-All: Train One Network and Specialize it for Efficient Deployment](https://mlsanity.com/paper/1908.09791/) (2019)
- [ProxylessNAS: Direct Neural Architecture Search on Target Task and Hardware](https://mlsanity.com/paper/1812.00332/) (2018)
- [Detecting Twenty-thousand Classes using Image-level Supervision](https://mlsanity.com/paper/2201.02605/) (2022)
- [DAB-DETR: Dynamic Anchor Boxes are Better Queries for DETR](https://mlsanity.com/paper/2201.12329/) (2022)
- [DETRs Beat YOLOs on Real-time Object Detection](https://mlsanity.com/paper/2304.08069/) (2023)
- [YOLO-World: Real-Time Open-Vocabulary Object Detection](https://mlsanity.com/paper/2401.17270/) (2024)
- [FBNet: Hardware-Aware Efficient ConvNet Design via Differentiable Neural Architecture Search](https://mlsanity.com/paper/1812.03443/) (2018)
- [Hybrid Task Cascade for Instance Segmentation](https://mlsanity.com/paper/1901.07518/) (2019)
- [YOLACT: Real-time Instance Segmentation](https://mlsanity.com/paper/1904.02689/) (2019)

## Built on by (most cited first, 30 of 75)

- [Ultralytics YOLO26: Unified Real-Time End-to-End Vision Models](https://mlsanity.com/paper/2606.03748/) (2026)
- [HERBench: A Benchmark for Multi-Evidence Integration in Video Question Answering](https://mlsanity.com/paper/2512.14870/) (2025)
- [QuPAINT: Physics-Aware Instruction Tuning Approach to Quantum Material Discovery](https://mlsanity.com/paper/2602.17478/) (2026)
- [EdgeCrafter: Compact ViTs for Edge Dense Prediction via Task-Specialized Distillation](https://mlsanity.com/paper/2603.18739/) (2026)
- [Class-specific diffusion models improve military object detection in a low-data domain](https://mlsanity.com/paper/2604.18076/) (2026)
- [YOLO26: A Comprehensive Architecture Overview and Key Improvements](https://mlsanity.com/paper/2602.14582/) (2026)
- [A Closer Look at Cross-Domain Few-Shot Object Detection: Fine-Tuning Matters and Parallel Decoder Helps](https://mlsanity.com/paper/2603.28182/) (2026)
- [LeAD-M3D: Leveraging Asymmetric Distillation for Real-Time Monocular 3D Detection](https://mlsanity.com/paper/2512.05663/) (2025)
- [EW-DETR: Evolving World Object Detection via Incremental Low-Rank DEtection TRansformer](https://mlsanity.com/paper/2602.20985/) (2026)
- [RiO-DETR: DETR for Real-time Oriented Object Detection](https://mlsanity.com/paper/2603.09411/) (2026)
- [Caries DETR: Tooth Structure-aware Prior and Lesion-aware Dynamic Loss Refinement for DETR Based Caries Detection](https://mlsanity.com/paper/2604.23718/) (2026)
- [Operationalizing Document AI: A Microservice Architecture for OCR and LLM Pipelines in Production](https://mlsanity.com/paper/2605.18818/) (2026)
- [AnyMo: Scaling Any-Modality Conditional Motion Generation with Masked Modeling](https://mlsanity.com/paper/2605.29488/) (2026)
- [RT-SDGOD: Real-Time Single-Domain Generalized Object Detection](https://mlsanity.com/paper/2606.09367/) (2026)
- [Falcon: Functional Assembly and Language for Compositional Reasoning in X-ray](https://mlsanity.com/paper/2606.25701/) (2026)
- [CP-WSP: A Declarative CP-SAT Framework for Configurable Multi-Constraint Workforce Scheduling](https://mlsanity.com/paper/2607.05177/) (2026)
- [SoccerNet 2026 Challenges Results](https://mlsanity.com/paper/2607.07320/) (2026)
- [A Review of Vision-Based Vehicle Detection for UAV-Based Traffic Monitoring: Experimental Insights and Future Directions](https://mlsanity.com/paper/2608.07571/) (2026)
- [HOIBlender: Blending Lightweight Detection with Vision-Language Priors for Efficient Human-Object Interaction Detection](https://mlsanity.com/paper/2609.23431/) (2026)
- [Cytoplasmic Strings Analysis in Human Embryo Time-Lapse Videos using Deep Learning Framework](https://mlsanity.com/paper/2512.09461/) (2025)
- [Comparing Object Detection Models for Electrical Substation Component Mapping](https://mlsanity.com/paper/2512.22454/) (2025)
- [PointRAFT: 3D deep learning for high-throughput prediction of potato tuber weight from partial point clouds](https://mlsanity.com/paper/2512.24193/) (2025)
- [PipeMFL-240K: A Large-scale Dataset and Benchmark for Object Detection in Pipeline Magnetic Flux Leakage Imaging](https://mlsanity.com/paper/2602.07044/) (2026)
- [Open-Vocabulary vs Supervised Learning Methods for Post-Disaster Visual Scene Understanding](https://mlsanity.com/paper/2603.01324/) (2026)
- [Show, Don't Tell: Detecting Novel Objects by Watching Human Videos](https://mlsanity.com/paper/2603.12751/) (2026)
- [MicroVision: An Open Dataset and Benchmark Models for Detecting Vulnerable Road Users and Micromobility Vehicles](https://mlsanity.com/paper/2603.18192/) (2026)
- [Defending against Patch-Based and Texture-Based Adversarial Attacks with Spectral Decomposition](https://mlsanity.com/paper/2604.10715/) (2026)
- [4th Workshop on Maritime Computer Vision (MaCVi): Challenge Overview](https://mlsanity.com/paper/2604.13244/) (2026)
- [ZoomSpec: A Physics-Guided Coarse-to-Fine Framework for Wideband Spectrum Sensing](https://mlsanity.com/paper/2604.13568/) (2026)
- [Accelerating New Product Introduction for Visual Quality Inspection via Few-Shot Diffusion-Based Defect Synthesis](https://mlsanity.com/paper/2604.22850/) (2026)

## Abstract (arXiv)

Open-vocabulary detectors achieve impressive performance on COCO, but often fail to generalize to real-world datasets with out-of-distribution classes not typically found in their pre-training. Rather than simply fine-tuning a heavy-weight vision-language model (VLM) for new domains, we introduce RF-DETR, a light-weight specialist detection transformer that discovers accuracy-latency Pareto curves for any target dataset with weight-sharing neural architecture search (NAS). Our approach fine-tunes a pre-trained base network on a target dataset and evaluates thousands of network configurations with different accuracy-latency tradeoffs without re-training. Further, we revisit the "tunable knobs" for NAS to improve the transferability of DETRs to diverse target domains. Notably, RF-DETR significantly improves over prior state-of-the-art real-time methods on COCO and Roboflow100-VL. RF-DETR (nano) achieves 48.0 AP on COCO, beating D-FINE (nano) by 5.3 AP at similar latency, and RF-DETR (2x-large) outperforms GroundingDINO (tiny) by 1.2 AP on Roboflow100-VL while running 20x as fast. To the best of our knowledge, RF-DETR (2x-large) is the first real-time detector to surpass 60 AP on COCO. Our code is available at https://github.com/roboflow/rf-detr