# YOLOE: Real-Time Seeing Anything

arXiv: https://arxiv.org/abs/2503.07465 · HTML: https://mlsanity.com/paper/2503.07465/

## Key facts

- Cited by 39 papers in the map (data through 14 Oct 2026).
- Builds on 67 of its 80 references that are in the map.
- First posted on arXiv 10 Mar 2025.

## Builds on (most cited first, 30 of 67)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [You Only Look Once: Unified, Real-Time Object Detection](https://mlsanity.com/paper/1506.02640/) (2015)
- [OPT: Open Pre-trained Transformer Language Models](https://mlsanity.com/paper/2205.01068/) (2022)
- [Deformable DETR: Deformable Transformers for End-to-End Object Detection](https://mlsanity.com/paper/2010.04159/) (2020)
- [Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection](https://mlsanity.com/paper/2303.05499/) (2023)
- [SSD: Single Shot MultiBox Detector](https://mlsanity.com/paper/1512.02325/) (2015)
- [Scaling Instruction-Finetuned Language Models](https://mlsanity.com/paper/2210.11416/) (2022)
- [Fast R-CNN](https://mlsanity.com/paper/1504.08083/) (2015)
- [SAM 2: Segment Anything in Images and Videos](https://mlsanity.com/paper/2408.00714/) (2024)
- [GLU Variants Improve Transformer](https://mlsanity.com/paper/2002.05202/) (2020)
- [YOLOv4: Optimal Speed and Accuracy of Object Detection](https://mlsanity.com/paper/2004.10934/) (2020)
- [FCOS: Fully Convolutional One-Stage Object Detection](https://mlsanity.com/paper/1904.01355/) (2019)
- [Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Models](https://mlsanity.com/paper/1505.04870/) (2015)
- [DINO: DETR with Improved DeNoising Anchor Boxes for End-to-End Object Detection](https://mlsanity.com/paper/2203.03605/) (2022)
- [Cascade R-CNN: Delving into High Quality Object Detection](https://mlsanity.com/paper/1712.00726/) (2017)
- [LVIS: A Dataset for Large Vocabulary Instance Segmentation](https://mlsanity.com/paper/1908.03195/) (2019)
- [Grounded Language-Image Pre-training](https://mlsanity.com/paper/2112.03857/) (2021)
- [YOLOv7: Trainable bag-of-freebies sets new state-of-the-art for real-time object detectors](https://mlsanity.com/paper/2207.02696/) (2022)
- [Open-vocabulary Object Detection via Vision and Language Knowledge Distillation](https://mlsanity.com/paper/2104.13921/) (2021)
- [MDETR -- Modulated Detection for End-to-End Multi-Modal Understanding](https://mlsanity.com/paper/2104.12763/) (2021)
- [Language-driven Semantic Segmentation](https://mlsanity.com/paper/2201.03546/) (2022)
- [RegionCLIP: Region-based Language-Image Pretraining](https://mlsanity.com/paper/2112.09106/) (2021)
- [Detecting Twenty-thousand Classes using Image-level Supervision](https://mlsanity.com/paper/2201.02605/) (2022)
- [Segment Everything Everywhere All at Once](https://mlsanity.com/paper/2304.06718/) (2023)

## Built on by (most cited first, 30 of 39)

- [What Is The Best 3D Scene Representation for Robotics? From Geometric to Foundation Models](https://mlsanity.com/paper/2512.03422/) (2025)
- [Few-Shot Adaptation of Grounding DINO for Agricultural Domain](https://mlsanity.com/paper/2504.07252/) (2025)
- [VLA-R: Vision-Language Action Retrieval toward Open-World End-to-End Autonomous Driving](https://mlsanity.com/paper/2511.12405/) (2025)
- [TRACE: Your Diffusion Model is Secretly an Instance Edge Detector](https://mlsanity.com/paper/2503.07982/) (2025)
- [OneVision: An End-to-End Generative Framework for Multi-view E-commerce Vision Search](https://mlsanity.com/paper/2510.05759/) (2025)
- [YOLOE-26: Integrating YOLO26 with YOLOE for Real-Time Open-Vocabulary Instance Segmentation](https://mlsanity.com/paper/2602.00168/) (2026)
- [OV-DEIM: Real-time DETR-Style Open-Vocabulary Object Detection with GridSynthetic Augmentation](https://mlsanity.com/paper/2603.07022/) (2026)
- [NoOVD: Novel Category Discovery and Embedding for Open-Vocabulary Object Detection](https://mlsanity.com/paper/2603.21069/) (2026)
- [PET-DINO: Unifying Visual Cues into Grounding DINO with Prompt-Enriched Training](https://mlsanity.com/paper/2604.00503/) (2026)
- [Instance-level Visual Active Tracking with Occlusion-Aware Planning](https://mlsanity.com/paper/2604.21453/) (2026)
- [CodeGraphVLP: Code-as-Planner Meets Semantic-Graph State for Non-Markovian Vision-Language-Action Models](https://mlsanity.com/paper/2604.22238/) (2026)
- [DOMR: Establishing Cross-View Segmentation via Dense Object Matching](https://mlsanity.com/paper/2508.04050/) (2025)
- [Plug-and-Play Clarifier: A Zero-Shot Multimodal Framework for Egocentric Intent Disambiguation](https://mlsanity.com/paper/2511.08971/) (2025)
- [Open-Text Aerial Detection: A Unified Framework For Aerial Visual Grounding And Detection](https://mlsanity.com/paper/2602.07827/) (2026)
- [Beyond Language: Grounding Referring Expressions with Hand Pointing in Egocentric Vision](https://mlsanity.com/paper/2603.26646/) (2026)
- [Auto-Labeling Data for Object Detection](https://mlsanity.com/paper/2506.02359/) (2025)
- [NOVA: Navigation via Object-Centric Visual Autonomy for High-Speed Target Tracking in Unstructured GPS-Denied Environments](https://mlsanity.com/paper/2506.18689/) (2025)
- [T-Rex: Task-Adaptive Spatial Representation Extraction for Robotic Manipulation with Vision-Language Models](https://mlsanity.com/paper/2506.19498/) (2025)
- [Progressive Bird's Eye View Perception for Safety-Critical Autonomous Driving: A Comprehensive Survey](https://mlsanity.com/paper/2508.07560/) (2025)
- [Designing Object Detection Models for TinyML: Foundations, Comparative Analysis, Challenges, and Emerging Solutions](https://mlsanity.com/paper/2508.08352/) (2025)
- [M3DMap: Object-aware Multimodal 3D Mapping for Dynamic Environments](https://mlsanity.com/paper/2508.17044/) (2025)
- [Semantic-Drive: Democratizing Long-Tail Data Curation via Open-Vocabulary Grounding and Neuro-Symbolic VLM Consensus](https://mlsanity.com/paper/2512.12012/) (2025)
- [User Prompting Strategies and Prompt Enhancement Methods for Open-Set Object Detection in XR Environments](https://mlsanity.com/paper/2601.23281/) (2026)
- [UniManip: General-Purpose Zero-Shot Robotic Manipulation with Agentic Operational Graph](https://mlsanity.com/paper/2602.13086/) (2026)
- [Prompt-Free Universal Region Proposal Network](https://mlsanity.com/paper/2603.17554/) (2026)
- [BakuFlow: A Streamlining Semi-Automatic Label Generation Tool](https://mlsanity.com/paper/2506.09083/) (2025)
- [ZERO: Industry-ready Vision Foundation Model with Multi-modal Prompts](https://mlsanity.com/paper/2507.04270/) (2025)
- [PRISM: Product Retrieval In Shopping Carts using Hybrid Matching](https://mlsanity.com/paper/2509.14985/) (2025)
- [Lattice Boltzmann Model for Learning Real-World Pixel Dynamicity](https://mlsanity.com/paper/2509.16527/) (2025)
- [Building Information Models to Robot-Ready Site Digital Twins (BIM2RDT): An Agentic AI Safety-First Framework](https://mlsanity.com/paper/2509.20705/) (2025)

## Abstract (arXiv)

Object detection and segmentation are widely employed in computer vision applications, yet conventional models like YOLO series, while efficient and accurate, are limited by predefined categories, hindering adaptability in open scenarios. Recent open-set methods leverage text prompts, visual cues, or prompt-free paradigm to overcome this, but often compromise between performance and efficiency due to high computational demands or deployment complexity. In this work, we introduce YOLOE, which integrates detection and segmentation across diverse open prompt mechanisms within a single highly efficient model, achieving real-time seeing anything. For text prompts, we propose Re-parameterizable Region-Text Alignment (RepRTA) strategy. It refines pretrained textual embeddings via a re-parameterizable lightweight auxiliary network and enhances visual-textual alignment with zero inference and transferring overhead. For visual prompts, we present Semantic-Activated Visual Prompt Encoder (SAVPE). It employs decoupled semantic and activation branches to bring improved visual embedding and accuracy with minimal complexity. For prompt-free scenario, we introduce Lazy Region-Prompt Contrast (LRPC) strategy. It utilizes a built-in large vocabulary and specialized embedding to identify all objects, avoiding costly language model dependency. Extensive experiments show YOLOE's exceptional zero-shot performance and transferability with high inference efficiency and low training cost. Notably, on LVIS, with 3× less training cost and 1.4× inference speedup, YOLOE-v8-S surpasses YOLO-Worldv2-S by 3.5 AP. When transferring to COCO, YOLOE-v8-L achieves 0.6 APb and 0.4 APm gains over closed-set YOLOv8-L with nearly 4× less training time. Code and models are available at https://github.com/THU-MIG/yoloe.