mlsanityData to 7 Oct 2026

Describe Anything: Detailed Localized Image and Video Captioning

arXiv 2504.16072 ↗First posted 22 Apr 2025cs.CV cs.AIcited by 84

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

Generating detailed and accurate descriptions for specific regions in images and videos remains a fundamental challenge for vision-language models. We introduce the Describe Anything Model (DAM), a model designed for detailed localized captioning (DLC). DAM preserves both local details and global context through two key innovations: a focal prompt, which ensures high-resolution encoding of targeted regions, and a localized vision backbone, which integrates precise localization with its broader context. To tackle the scarcity of high-quality DLC data, we propose a Semi-supervised learning (SSL)-based Data Pipeline (DLC-SDP). DLC-SDP starts with existing segmentation datasets and expands to unlabeled web images using SSL. We introduce DLC-Bench, a benchmark designed to evaluate DLC without relying on reference captions. DAM sets new state-of-the-art on 7 benchmarks spanning keyword-level, phrase-level, and detailed multi-sentence localized image and video captioning.

Key facts. Cited by 84 papers in the map (data through 7 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 75 of its 98 references that are in the map. First posted on arXiv 22 Apr 2025.

Builds on

75 of 98 references.

Closest ancestors · 7

Same direction, newest first

  1. 2024PaliGemma: A versatile 3B VLM for transfer10 Jul
  2. 2024Revisiting Referring Expression Comprehension Evaluation in the Era of Large Multimodal Models24 Jun
  3. 2024RegionGPT: Towards Region Understanding Vision Language Model4 Mar
  4. 2023GLaMM: Pixel Grounding Large Multimodal Model6 Nov
  5. 2023Ferret: Refer and Ground Anything Anywhere at Any Granularity11 Oct
  6. 2022Flamingo: a Visual Language Model for Few-Shot Learning29 Apr
  7. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation28 Jan

Other references · 68

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2024The Llama 3 Herd of Modelscited by 14,192
  4. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  5. 2023Segment Anythingcited by 8,158
  6. 2023Visual Instruction Tuningcited by 7,792
  7. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  8. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,151
  9. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,274
  10. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
  11. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,048
  12. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
38 more
  1. 2024SAM 2: Segment Anything in Images and Videoscited by 2,784
  2. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  3. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  4. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,534
  5. 2020FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidencecited by 2,080
  6. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,027
  7. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scalingcited by 1,573
  8. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,447
  9. 2019MixMatch: A Holistic Approach to Semi-Supervised Learningcited by 1,390
  10. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,352
  11. 2019Self-training with Noisy Student improves ImageNet classificationcited by 1,304
  12. 2019LVIS: A Dataset for Large Vocabulary Instance Segmentationcited by 1,215
  13. 2019Unsupervised Data Augmentation for Consistency Trainingcited by 1,057
  14. 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captionscited by 1,000
  15. 2022Token Merging: Your ViT But Fastercited by 995
  16. 2015Generation and Comprehension of Unambiguous Object Descriptionscited by 897
  17. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magiccited by 891
  18. 2016COCO-Stuff: Thing and Stuff Classes in Contextcited by 858
  19. 2016SPICE: Semantic Propositional Image Caption Evaluationcited by 820
  20. 2023GLIGEN: Open-Set Grounded Text-to-Image Generationcited by 774
  21. 2022Exploring Plain Vision Transformer Backbones for Object Detectioncited by 723
  22. 2023VILA: On Pre-training for Visual Language Modelscited by 686
  23. 2023The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision)cited by 677
  24. 2021Common Objects in 3D: Large-Scale Learning and Evaluation of Real-life 3D Category Reconstructioncited by 646
  25. 2021RegionCLIP: Region-based Language-Image Pretrainingcited by 612
  26. 2023V*: Guided Visual Search as a Core Mechanism in Multimodal LLMscited by 459
  27. 2019Cascade R-CNN: High Quality Object Detection and Instance Segmentationcited by 428
  28. 2024Panda-70M: Captioning 70M Videos with Multiple Cross-Modality Teacherscited by 389
  29. 2023EVA-02: A Visual Representation for Neon Genesiscited by 377
  30. 2019ReMixMatch: Semi-Supervised Learning with Distribution Alignment and Augmentation Anchoringcited by 349
  31. 2023Scaling Open-Vocabulary Object Detectioncited by 335
  32. 2023OBELICS: An Open Web-Scale Filtered Dataset of Interleaved Image-Text Documentscited by 316
  33. 2023GPT4RoI: Instruction Tuning Large Language Model on Region-of-Interestcited by 300
  34. 2021Focal Self-attention for Local-Global Interactions in Vision Transformerscited by 286
  35. 2024Enhancing the Reasoning Ability of Multimodal Large Language Models via Mixed Preference Optimizationcited by 238
  36. 2023MOSE: A New Dataset for Video Object Segmentation in Complex Scenescited by 232
  37. 2023Osprey: Pixel Understanding with Visual Instruction Tuningcited by 158
  38. 2023Alpha-CLIP: A CLIP Model Focusing on Wherever You Wantcited by 150

50 most cited of 68 All 68 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 39; 2026: 45.

84 papers.

Sort:

By area:

  1. 2025OpenVE-3M: A Large-Scale High-Quality Dataset for Instruction-Guided Video Editingcited by 50
  2. 2025Perceive Anything: Recognize, Explain, Caption, and Segment Anything in Images and Videoscited by 49
  3. 2025ViCrit: A Verifiable Reinforcement Learning Proxy Task for Visual Perception in VLMscited by 25
  4. 2025PixelRefer: A Unified Framework for Spatio-Temporal Object Referring with Arbitrary Granularitycited by 21
  5. 2025Describe Anything Anywhere At Any Momentcited by 19
  6. 2025Grasp Any Region: Towards Precise, Contextual Pixel Understanding for Multimodal LLMscited by 18
  7. 2025CaptionQA: Is Your Caption as Useful as the Image Itself?cited by 14
  8. 2026Multiplayer Interactive World Models with Representation Autoencoderscited by 14
  9. 2025Object-centric Video Question Answering with Visual Grounding and Referringcited by 12
  10. 2025VLM-FO1: Bridging the Gap Between High-Level Reasoning and Fine-Grained Perception in VLMscited by 12
  11. 2025RynnEC: Bringing MLLMs into Embodied Worldcited by 11
  12. 2025WeDetect: Fast Open-Vocabulary Object Detection as Retrievalcited by 11
38 more
  1. 2026Thinking in Dynamics: How Multimodal Large Language Models Perceive, Track, and Reason Dynamics in Physical 4D Worldcited by 11
  2. 2025Web2Grasp: Learning Functional Grasps from Web Images of Hand-Object Interactionscited by 10
  3. 2025Describe Anything in Medical Imagescited by 10
  4. 2026SAMTok: Representing Any Mask with Two Wordscited by 10
  5. 2025Dense360: Dense Understanding from Omnidirectional Panoramascited by 9
  6. 2025SAMA: Towards Multi-Turn Referential Grounded Video Chat with Large Language Modelscited by 8
  7. 2025IF-VidCap: Can Video Caption Models Follow Instructions?cited by 7
  8. 2025Unified Embodied VLM Reasoning with Robotic Action via Autoregressive Discretized Pre-trainingcited by 7
  9. 2025Strefer: Empowering Video LLMs with Space-Time Referring and Reasoning via Synthetic Instruction Datacited by 6
  10. 2026TextShield-R1: Reinforced Reasoning for Tampered Text Detectioncited by 6
  11. 2026LMMs Meet Object-Centric Vision: Understanding, Segmentation, Editing and Generationcited by 6
  12. 2025Highlighting What Matters: Promptable Embeddings for Attribute-Focused Image Retrievalcited by 5
  13. 2025Describe Anything Model for Visual Question Answering on Text-rich Imagescited by 5
  14. 2025DescribeEarth: Describe Anything for Remote Sensing Imagescited by 5
  15. 2025DynamicVerse: A Physically-Aware Multimodal Framework for 4D World Modelingcited by 5
  16. 2026OpenVoxel: Training-Free Grouping and Captioning Voxels for Open-Vocabulary 3D Scene Understandingcited by 5
  17. 2025PositionIC: Unified Position and Identity Consistency for Image Customizationcited by 4
  18. 2025VDC-Agent: When Video Detailed Captioners Evolve Themselves via Agentic Self-Reflectioncited by 4
  19. 2026Steerable Visual Representationscited by 4
  20. 2026MetaPoint: Unlocking Precise Spatial Control in Agentic Visual Generationcited by 4
  21. 2025PETAR: Localized Findings Generation with Mask-Aware Vision-Language Modeling for PET Automated Reportingcited by 3
  22. 2025Black-Box Membership Inference Attack for LVLMs via Prior Knowledge-Calibrated Memory Probingcited by 3
  23. 2026Synthetic Visual Genome 2: Extracting Large-scale Spatio-Temporal Scene Graphs from Videoscited by 3
  24. 2026Building a Precise Video Language with Human-AI Oversightcited by 3
  25. 2026ClaimDiff-RL: Fine-Grained Caption Reinforcement Learning through Visual Claim Comparisoncited by 3
  26. 2026Remember with Confidence: Uncertainty Quantification for Spatio-temporal Memory with Probabilistic Guaranteescited by 3
  27. 2025FOCUS: Unified Vision-Language Modeling for Interactive Editing Driven by Referential Segmentationcited by 2
  28. 2025AVC-DPO: Aligned Video Captioning via Direct Preference Optimizationcited by 2
  29. 2025CLAW: A Vision-Language-Action Framework for Weight-Aware Robotic Graspingcited by 2
  30. 2025Video Object Segmentation-Aware Audio Generationcited by 2
  31. 2025VIRTUE: Visual-Interactive Text-Image Universal Embeddercited by 2
  32. 2025UFVideo: Towards Unified Fine-Grained Video Cooperative Understanding with Large Language Modelscited by 2
  33. 2026Images in Sentences: Scaling Interleaved Instructions for Unified Visual Generationcited by 2
  34. 2025MetaCaptioner: Towards Generalist Visual Captioning with Open-source Suitescited by 1
  35. 2025Towards Accurate UAV Image Perception: Guiding Vision-Language Models with Stronger Task Promptscited by 1
  36. 2025M3-Verse: A "Spot the Difference" Challenge for Large Multimodal Modelscited by 1
  37. 2026VersaViT: Enhancing MLLM Vision Backbones via Task-Guided Optimizationcited by 1
  38. 2026TraceVision: Trajectory-Aware Vision-Language Model for Human-Like Spatial Understandingcited by 1

50 most cited of 84 All 84 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023Rewrite Caption Semantics: Bridging Semantic Gaps for Language-Supervised Semantic Segmentation
  2. 2024FlexCap: Describe Anything in Images in Controllable Detail
  3. 2024Graph-Based Captioning: Enhancing Visual Descriptions by Interconnecting Region Captions
  4. 2025Caption Anything in Video: Fine-grained Object-centric Captioning via Spatiotemporal Multimodal Prompting
  5. 2024Benchmarking Large Vision-Language Models via Directed Scene Graph for Comprehensive Image Captioning
  6. 2023Zero-Shot Dense Video Captioning by Jointly Optimizing Text and Moment
  7. 2024Benchmarking and Improving Detail Image Caption
  8. 2023Grounding Everything: Emerging Localization Properties in Vision-Language Transformers