# DINO-X: A Unified Vision Model for Open-World Object Detection and Understanding

arXiv: https://arxiv.org/abs/2411.14347 · HTML: https://mlsanity.com/paper/2411.14347/

## Key facts

- Cited by 89 papers in the map (data through 14 Oct 2026).
- Builds on 75 of its 79 references that are in the map.
- First posted on arXiv 21 Nov 2024.

## Builds on (most cited first, 30 of 75)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [The Power of Scale for Parameter-Efficient Prompt Tuning](https://mlsanity.com/paper/2104.08691/) (2021)
- [OPT: Open Pre-trained Transformer Language Models](https://mlsanity.com/paper/2205.01068/) (2022)
- [Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection](https://mlsanity.com/paper/2303.05499/) (2023)
- [SAM 2: Segment Anything in Images and Videos](https://mlsanity.com/paper/2408.00714/) (2024)
- [Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations](https://mlsanity.com/paper/1602.07332/) (2016)
- [Masked-attention Mask Transformer for Universal Image Segmentation](https://mlsanity.com/paper/2112.01527/) (2021)
- [Generalized Intersection over Union: A Metric and A Loss for Bounding Box Regression](https://mlsanity.com/paper/1902.09630/) (2019)
- [Deep High-Resolution Representation Learning for Human Pose Estimation](https://mlsanity.com/paper/1902.09212/) (2019)
- [DINO: DETR with Improved DeNoising Anchor Boxes for End-to-End Object Detection](https://mlsanity.com/paper/2203.03605/) (2022)
- [LVIS: A Dataset for Large Vocabulary Instance Segmentation](https://mlsanity.com/paper/1908.03195/) (2019)
- [Grounded Language-Image Pre-training](https://mlsanity.com/paper/2112.03857/) (2021)
- [Kosmos-2: Grounding Multimodal Large Language Models to the World](https://mlsanity.com/paper/2306.14824/) (2023)
- [Grounded SAM: Assembling Open-World Models for Diverse Visual Tasks](https://mlsanity.com/paper/2401.14159/) (2024)
- [Shikra: Unleashing Multimodal LLM's Referential Dialogue Magic](https://mlsanity.com/paper/2306.15195/) (2023)
- [Generation and Comprehension of Unambiguous Object Descriptions](https://mlsanity.com/paper/1511.02283/) (2015)
- [MDETR -- Modulated Detection for End-to-End Multi-Modal Understanding](https://mlsanity.com/paper/2104.12763/) (2021)
- [Ferret: Refer and Ground Anything Anywhere at Any Granularity](https://mlsanity.com/paper/2310.07704/) (2023)
- [YOLO-World: Real-Time Open-Vocabulary Object Detection](https://mlsanity.com/paper/2401.17270/) (2024)
- [Simple Baselines for Human Pose Estimation and Tracking](https://mlsanity.com/paper/1804.06208/) (2018)
- [ViTPose: Simple Vision Transformer Baselines for Human Pose Estimation](https://mlsanity.com/paper/2204.12484/) (2022)
- [PointRend: Image Segmentation as Rendering](https://mlsanity.com/paper/1912.08193/) (2019)
- [End-to-End Human Pose and Mesh Reconstruction with Transformers](https://mlsanity.com/paper/2012.09760/) (2020)
- [EVA-02: A Visual Representation for Neon Genesis](https://mlsanity.com/paper/2303.11331/) (2023)
- [Mask DINO: Towards A Unified Transformer-based Framework for Object Detection and Segmentation](https://mlsanity.com/paper/2206.02777/) (2022)

## Built on by (most cited first, 30 of 89)

- [Detect Anything via Next Point Prediction](https://mlsanity.com/paper/2510.12798/) (2025)
- [OctoTools: An Agentic Framework with Extensible Tools for Complex Reasoning](https://mlsanity.com/paper/2502.11271/) (2025)
- [Toward Visual Grounding: A Survey](https://mlsanity.com/paper/2412.20206/) (2024)
- [VisionReasoner: Unified Reasoning-Integrated Visual Perception via Reinforcement Learning](https://mlsanity.com/paper/2505.12081/) (2025)
- [Articulate AnyMesh: Open-Vocabulary 3D Articulated Objects Modeling](https://mlsanity.com/paper/2502.02590/) (2025)
- [Image Generators are Generalist Vision Learners](https://mlsanity.com/paper/2604.20329/) (2026)
- [DICEPTION: A Generalist Diffusion Model for Visual Perceptual Tasks](https://mlsanity.com/paper/2502.17157/) (2025)
- [YOLOE: Real-Time Seeing Anything](https://mlsanity.com/paper/2503.07465/) (2025)
- [Rex-Thinker: Grounded Object Referring via Chain-of-Thought Reasoning](https://mlsanity.com/paper/2506.04034/) (2025)
- [DynamicEarth: How Far are We from Open-Vocabulary Change Detection?](https://mlsanity.com/paper/2501.12931/) (2025)
- [Referring to Any Person](https://mlsanity.com/paper/2503.08507/) (2025)
- [ObjectVLA: End-to-End Open-World Object Manipulation Without Demonstration](https://mlsanity.com/paper/2502.19250/) (2025)
- [RefDrone: A Challenging Benchmark for Referring Expression Comprehension in Drone Scenes](https://mlsanity.com/paper/2502.00392/) (2025)
- [SpaceVista: All-Scale Visual Spatial Reasoning from mm to km](https://mlsanity.com/paper/2510.09606/) (2025)
- [TiViBench: Benchmarking Think-in-Video Reasoning for Video Generative Models](https://mlsanity.com/paper/2511.13704/) (2025)
- [Parallels Between VLA Model Post-Training and Human Motor Learning: Progress, Challenges, and Trends](https://mlsanity.com/paper/2506.20966/) (2025)
- [Easier Said Than Done: Unpacking Intent-Behavior Gap in Jailbreaking LLM-Based Robots](https://mlsanity.com/paper/2412.16633/) (2024)
- [WeDetect: Fast Open-Vocabulary Object Detection as Retrieval](https://mlsanity.com/paper/2512.12309/) (2025)
- [VLM-FO1: Bridging the Gap Between High-Level Reasoning and Fine-Grained Perception in VLMs](https://mlsanity.com/paper/2509.25916/) (2025)
- [DeFacto: Counterfactual Thinking with Images for Enforcing Evidence-Grounded and Faithful Reasoning](https://mlsanity.com/paper/2509.20912/) (2025)
- [PhysToolBench: Benchmarking Physical Tool Understanding for MLLMs](https://mlsanity.com/paper/2510.09507/) (2025)
- [LED: LLM Enhanced Open-Vocabulary Object Detection without Human Curated Data Generation](https://mlsanity.com/paper/2503.13794/) (2025)
- [InstructSAM: A Training-Free Framework for Instruction-Oriented Remote Sensing Object Recognition](https://mlsanity.com/paper/2505.15818/) (2025)
- [MV-CoLight: Efficient Object Compositing with Consistent Lighting and Shadow Generation](https://mlsanity.com/paper/2505.21483/) (2025)
- [LVM4CSI: Enabling Direct Application of Pre-Trained Large Vision Models for Wireless Channel Tasks](https://mlsanity.com/paper/2507.05121/) (2025)
- [Dynamic-DINO: Fine-Grained Mixture of Experts Tuning for Real-time Open-Vocabulary Object Detection](https://mlsanity.com/paper/2507.17436/) (2025)
- [Mind to Hand: Purposeful Robotic Control via Embodied Reasoning](https://mlsanity.com/paper/2512.08580/) (2025)
- [RC-NF: Robot-Conditioned Normalizing Flow for Real-Time Anomaly Detection in Robotic Manipulation](https://mlsanity.com/paper/2603.11106/) (2026)
- [DINO-R1: Incentivizing Reasoning Capability in Vision Foundation Models](https://mlsanity.com/paper/2505.24025/) (2025)
- [MTPano: Multi-Task Panoramic Scene Understanding via Label-Free Integration of Dense Prediction Priors](https://mlsanity.com/paper/2602.05330/) (2026)

## Abstract (arXiv)

In this paper, we introduce DINO-X, which is a unified object-centric vision model developed by IDEA Research with the best open-world object detection performance to date. DINO-X employs the same Transformer-based encoder-decoder architecture as Grounding DINO 1.5 to pursue an object-level representation for open-world object understanding. To make long-tailed object detection easy, DINO-X extends its input options to support text prompt, visual prompt, and customized prompt. With such flexible prompt options, we develop a universal object prompt to support prompt-free open-world detection, making it possible to detect anything in an image without requiring users to provide any prompt. To enhance the model's core grounding capability, we have constructed a large-scale dataset with over 100 million high-quality grounding samples, referred to as Grounding-100M, for advancing the model's open-vocabulary detection performance. Pre-training on such a large-scale grounding dataset leads to a foundational object-level representation, which enables DINO-X to integrate multiple perception heads to simultaneously support multiple object perception and understanding tasks, including detection, segmentation, pose estimation, object captioning, object-based QA, etc. Experimental results demonstrate the superior performance of DINO-X. Specifically, the DINO-X Pro model achieves 56.0 AP, 59.8 AP, and 52.4 AP on the COCO, LVIS-minival, and LVIS-val zero-shot object detection benchmarks, respectively. Notably, it scores 63.3 AP and 56.5 AP on the rare classes of LVIS-minival and LVIS-val benchmarks, improving the previous SOTA performance by 5.8 AP and 5.0 AP. Such a result underscores its significantly improved capacity for recognizing long-tailed objects.