mlsanityData to 14 Oct 2026

DenseWorld-1M: Towards Detailed Dense Grounded Caption in the Real World

arXiv 2506.24102 ↗First posted 30 Jun 2025cs.CVcited by 16

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

Multimodal Large Language Models (MLLMs) demonstrate a complex understanding of scenes, benefiting from large-scale and high-quality datasets. Most existing caption datasets lack the ground locations and relations for visual entities. Several grounded caption datasets face the problems of missing detailed descriptions, relations, and massive object descriptions on high-resolution images. To fill this gap for the community, we present DenseWorld-1M, the first massive, detailed, dense grounded caption dataset in the real world. We design a three-stage labeling pipeline, containing open-world perception, detailed object caption generation, and dense caption merging. The first stage obtains entity-level masks and labels. The second stage generates the object-level, detailed captions with the guidance of masks and labels from the first stage. The final stage merges object captions and masks into spatial and relational dense captions. To accelerate the labeling process and improve caption quality, we present two VLM models: the Detailed Region Caption model and the Spatial Caption Merging model. Extensive experiments on various settings, including vision-language understanding, visual grounding, and region caption generation, demonstrate the effectiveness of our DenseWorld-1M dataset and labeling models.

Key facts. Cited by 16 papers in the map (data through 14 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 86 of its 99 references that are in the map. First posted on arXiv 30 Jun 2025.

Builds on

86 of 99 references.

Closest ancestors · 8

Same direction, newest first

  1. 2025Seed1.5-VL Technical Report11 May
  2. 2025Sa2VA: Marrying SAM2 with MLLM for Dense Grounded Understanding of Images and Videos7 Jan
  3. 2024World to Code: Multi-modal Data Generation via Self-Instructed Compositional Captioning and Filtering30 Sep
  4. 2024Are We on the Right Way for Evaluating Large Vision-Language Models?29 Mar
  5. 2024Mini-Gemini: Mining the Potential of Multi-modality Vision Language Models27 Mar
  6. 2024ALLaVA: Harnessing GPT4V-Synthesized Data for Lite Vision-Language Models18 Feb
  7. 2023PixelLM: Pixel Reasoning with Large Multimodal Model4 Dec
  8. 2023GLaMM: Pixel Grounding Large Multimodal Model6 Nov

Other references · 78

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2025Qwen3 Technical Reportcited by 8,820
  3. 2023Segment Anythingcited by 8,234
  4. 2023Visual Instruction Tuningcited by 7,869
  5. 2020End-to-End Object Detection with Transformerscited by 7,414
  6. 2017Mask R-CNNcited by 7,277
  7. 2025Qwen2.5-VL Technical Reportcited by 5,587
  8. 2024GPT-4o System Cardcited by 4,588
  9. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
  10. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,234
  11. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,152
  12. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,332
38 more
  1. 2016DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFscited by 3,241
  2. 2015Fast R-CNNcited by 2,933
  3. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,656
  4. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,141
  5. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,080
  6. 2025InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Modelscited by 1,834
  7. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,652
  8. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scalingcited by 1,599
  9. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,462
  10. 2023Video-LLaVA: Learning United Visual Representation by Alignment Before Projectioncited by 1,402
  11. 2020DocVQA: A Dataset for VQA on Document Imagescited by 1,317
  12. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,216
  13. 2016Modeling Context in Referring Expressionscited by 1,063
  14. 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captionscited by 1,005
  15. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehensioncited by 940
  16. 2015Generation and Comprehension of Unambiguous Object Descriptionscited by 904
  17. 2024Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMscited by 838
  18. 2016A Diagram Is Worth A Dozen Imagescited by 804
  19. 2018Panoptic Segmentationcited by 802
  20. 2017Dense-Captioning Events in Videoscited by 793
  21. 2024Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMscited by 735
  22. 2023GenImage: A Million-Scale Benchmark for Detecting AI-Generated Imagecited by 321
  23. 2022Images Speak in Images: A Generalist Painter for In-Context Visual Learningcited by 318
  24. 2023Recognize Anything: A Strong Image Tagging Modelcited by 309
  25. 2024MME-RealWorld: Could Your Multimodal LLM Challenge High-Resolution Real-World Scenarios that are Difficult for Humans?cited by 292
  26. 2025VITA-1.5: Towards GPT-4o Level Real-Time Vision and Speech Interactioncited by 197
  27. 2023Osprey: Pixel Understanding with Visual Instruction Tuningcited by 159
  28. 2023RAIN: Your Language Models Can Align Themselves without Finetuningcited by 157
  29. 2024Video-RAG: Visually-aligned Retrieval-Augmented Long Video Comprehensioncited by 138
  30. 2022Open-Vocabulary Universal Image Segmentation with MaskCLIPcited by 132
  31. 2024DOCCI: Descriptions of Connected and Contrasting Imagescited by 132
  32. 2020Semantic Flow for Fast and Accurate Scene Parsingcited by 128
  33. 2024OMG-LLaVA: Bridging Image-level, Object-level, Pixel-level Reasoning and Understandingcited by 127
  34. 2024VISA: Reasoning Video Object Segmentation via Large Language Modelscited by 127
  35. 2024MAmmoTH-VL: Eliciting Multimodal Reasoning with Instruction Tuning at Scalecited by 127
  36. 2023Transformer-Based Visual Segmentation: A Surveycited by 124
  37. 2023CLIPSelf: Vision Transformer Distills Itself for Open-Vocabulary Dense Predictioncited by 118
  38. 2023The All-Seeing Project: Towards Panoptic Visual Recognition and Understanding of the Open Worldcited by 116

50 most cited of 78 All 78 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 8; 2026: 8.

16 papers.

Sort:

By area:

  1. 2025Kwai Keye-VL 1.5 Technical Reportcited by 73
  2. 2025Kwai Keye-VL Technical Reportcited by 47
  3. 2025Grasp Any Region: Towards Precise, Contextual Pixel Understanding for Multimodal LLMscited by 18
  4. 2025Visual Reasoning Tracer: Object-Level Grounded Reasoning Benchmarkcited by 11
  5. 2025GroundingME: Exposing the Visual Grounding Gap in MLLMs through Multi-Dimensional Evaluationcited by 9
  6. 2026RubiCap: Rubric-Guided Reinforcement Learning for Dense Image Captioningcited by 9
  7. 2026Perceptio: Perception Enhanced Vision Language Models via Spatial Token Generationcited by 3
  8. 2026Let ViT Speak: Generative Language-Image Pre-trainingcited by 2
  9. 2025SVAC: Scaling Is All You Need For Referring Video Object Segmentationcited by 1
  10. 2025MetaCaptioner: Towards Generalist Visual Captioning with Open-source Suitescited by 1
  11. 2025Differences That Matter: Auditing Models for Capability Gap Discovery and Rectificationcited by 1
  12. 2026PerceptionDLM: Parallel Region Perception with Multimodal Diffusion Language Modelscited by 1
4 more
  1. 2026MotionAtlas: Detailed Region Captioning for Motion-Centric Videoscited by 1
  2. 2026Actor as Its Own Critic: Unifying Region Understanding and Localization via CycleGRPOcited by 1
  3. 2026CapProbe: Evaluating Detailed Image Captions via Full-Scene Dense Question Answeringcited by 0
  4. 2026MultiModal Code-Switching: Interleaving Visual Objects into Language for Explicit Object-Level Alignmentcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023A Picture is Worth More Than 77 Text Tokens: Evaluating CLIP-Style Models on Dense Captions
  2. 2026PANORAMA: Panoptic Grounded Captioning via Mask Proposal Selection
  3. 2024Benchmarking Large Vision-Language Models via Directed Scene Graph for Comprehensive Image Captioning
  4. 2023CapDet: Unifying Dense Captioning and Open-World Detection Pretraining
  5. 2024BLIP3-KALE: Knowledge Augmented Large-Scale Dense Captions
  6. 2025Generating Accurate and Detailed Captions for High-Resolution Images
  7. 2025DenseAnnotate: Enabling Scalable Dense Caption Collection for Images and 3D Scenes via Spoken Descriptions
  8. 2025CAPability: A Comprehensive Visual Caption Benchmark for Evaluating Both Correctness and Thoroughness