# COCO-Text: Dataset and Benchmark for Text Detection and Recognition in Natural Images

arXiv: https://arxiv.org/abs/1601.07140 · HTML: https://mlsanity.com/paper/1601.07140/

## Key facts

- Cited by 236 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 26 Jan 2016.

## Built on by (most cited first, 30 of 236)

- [Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling](https://mlsanity.com/paper/2412.05271/) (2024)
- [How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suites](https://mlsanity.com/paper/2404.16821/) (2024)
- [DeepSeek-VL: Towards Real-World Vision-Language Understanding](https://mlsanity.com/paper/2403.05525/) (2024)
- [VideoLLaMA 3: Frontier Multimodal Foundation Models for Image and Video Understanding](https://mlsanity.com/paper/2501.13106/) (2025)
- [OCRBench: On the Hidden Mystery of OCR in Large Multimodal Models](https://mlsanity.com/paper/2305.07895/) (2023)
- [Unified-IO 2: Scaling Autoregressive Multimodal Models with Vision, Language, Audio, and Action](https://mlsanity.com/paper/2312.17172/) (2023)
- [LVLM-eHub: A Comprehensive Evaluation Benchmark for Large Vision-Language Models](https://mlsanity.com/paper/2306.09265/) (2023)
- [General OCR Theory: Towards OCR-2.0 via a Unified End-to-end Model](https://mlsanity.com/paper/2409.01704/) (2024)
- [An Introduction to Vision-Language Modeling](https://mlsanity.com/paper/2405.17247/) (2024)
- [TextMonkey: An OCR-Free Large Multimodal Model for Understanding Document](https://mlsanity.com/paper/2403.04473/) (2024)
- [OCRBench v2: An Improved Benchmark for Evaluating Large Multimodal Models on Visual Text Localization and Reasoning](https://mlsanity.com/paper/2501.00321/) (2024)
- [TextOCR: Towards large-scale end-to-end reasoning for arbitrary-shaped scene text](https://mlsanity.com/paper/2105.05486/) (2021)
- [Scene Text Recognition with Permuted Autoregressive Sequence Models](https://mlsanity.com/paper/2207.06966/) (2022)
- [PACO: Parts and Attributes of Common Objects](https://mlsanity.com/paper/2301.01795/) (2023)
- [ImageBind-LLM: Multi-modality Instruction Tuning](https://mlsanity.com/paper/2309.03905/) (2023)
- [AuroraCap: Efficient, Performant Video Detailed Captioning and a New Benchmark](https://mlsanity.com/paper/2410.03051/) (2024)
- [M3IT: A Large-Scale Dataset towards Multi-Modal Multilingual Instruction Tuning](https://mlsanity.com/paper/2306.04387/) (2023)
- [NVLM: Open Frontier-Class Multimodal LLMs](https://mlsanity.com/paper/2409.11402/) (2024)
- [MultiInstruct: Improving Multi-Modal Zero-Shot Learning via Instruction Tuning](https://mlsanity.com/paper/2212.10773/) (2022)
- [EmbSpatial-Bench: Benchmarking Spatial Understanding for Embodied Tasks with Large Vision-Language Models](https://mlsanity.com/paper/2406.05756/) (2024)
- [Draw-and-Understand: Leveraging Visual Prompts to Enable MLLMs to Comprehend What You Want](https://mlsanity.com/paper/2403.20271/) (2024)
- [Nemotron-H: A Family of Accurate and Efficient Hybrid Mamba-Transformer Models](https://mlsanity.com/paper/2504.03624/) (2025)
- [Mini-InternVL: A Flexible-Transfer Pocket Multimodal Model with 5% Parameters and 90% Performance](https://mlsanity.com/paper/2410.16261/) (2024)
- [ABCNet v2: Adaptive Bezier-Curve Network for Real-time End-to-end Text Spotting](https://mlsanity.com/paper/2105.03620/) (2021)
- [Vary: Scaling up the Vision Vocabulary for Large Vision-Language Models](https://mlsanity.com/paper/2312.06109/) (2023)
- [Mono-InternVL: Pushing the Boundaries of Monolithic Multimodal Large Language Models with Endogenous Visual Pre-training](https://mlsanity.com/paper/2410.08202/) (2024)
- [Tarsier2: Advancing Large Vision-Language Models from Detailed Video Description to Comprehensive Video Understanding](https://mlsanity.com/paper/2501.07888/) (2025)
- [A Survey of Multimodal Large Language Model from A Data-centric Perspective](https://mlsanity.com/paper/2405.16640/) (2024)
- [LaTr: Layout-Aware Transformer for Scene-Text VQA](https://mlsanity.com/paper/2112.12494/) (2021)
- [Eagle 2: Building Post-Training Data Strategies from Scratch for Frontier Vision-Language Models](https://mlsanity.com/paper/2501.14818/) (2025)

## Abstract (arXiv)

This paper describes the COCO-Text dataset. In recent years large-scale datasets like SUN and Imagenet drove the advancement of scene understanding and object recognition. The goal of COCO-Text is to advance state-of-the-art in text detection and recognition in natural images. The dataset is based on the MS COCO dataset, which contains images of complex everyday scenes. The images were not collected with text in mind and thus contain a broad variety of text instances. To reflect the diversity of text in natural scenes, we annotate text with (a) location in terms of a bounding box, (b) fine-grained classification into machine printed text and handwritten text, (c) classification into legible and illegible text, (d) script of the text and (e) transcriptions of legible text. The dataset contains over 173k text annotations in over 63k images. We provide a statistical analysis of the accuracy of our annotations. In addition, we present an analysis of three leading state-of-the-art photo Optical Character Recognition (OCR) approaches on our dataset. While scene text detection and recognition enjoys strong advances in recent years, we identify significant shortcomings motivating future work.