mlsanityData to 7 Oct 2026

Caption Anything: Interactive Image Description with Diverse Multimodal Controls

arXiv 2305.02677 ↗First posted 4 May 2023cs.CVcited by 98

Not on the map. Why?

Cited by 35 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Controllable image captioning is an emerging multimodal topic that aims to describe the image with natural language following human purpose, textite.g., looking at the specified regions or telling in a particular text style. State-of-the-art methods are trained on annotated pairs of input controls and output captions. However, the scarcity of such well-annotated multimodal data largely limits their usability and scalability for interactive AI systems. Leveraging unimodal instruction-following foundation models is a promising alternative that benefits from broader sources of data. In this paper, we present Caption AnyThing (CAT), a foundation model augmented image captioning framework supporting a wide range of multimodel controls: 1) visual controls, including points, boxes, and trajectories; 2) language controls, such as sentiment, length, language, and factuality. Powered by Segment Anything Model (SAM) and ChatGPT, we unify the visual and language prompts into a modularized framework, enabling the flexible combination between different controls. Extensive case studies demonstrate the user intention alignment capabilities of our framework, shedding light on effective user interaction modeling in vision-language applications. Our code is publicly available at https://github.com/ttengwang/Caption-Anything.

Key facts. Cited by 98 papers in the map (data through 7 Oct 2026). Builds on 38 of its 49 references that are in the map. First posted on arXiv 4 May 2023.

Builds on

38 of 49 references.

References · 38

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2022Training language models to follow instructions with human feedbackcited by 15,712
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  5. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  6. 2023Segment Anythingcited by 8,158
  7. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  8. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  9. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  10. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  11. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  12. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
26 more
  1. 2015Show, Attend and Tell: Neural Image Caption Generation with Visual Attentioncited by 1,421
  2. 2017Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answeringcited by 1,310
  3. 2014Show and Tell: A Neural Image Caption Generatorcited by 877
  4. 2022Galactica: A Large Language Model for Sciencecited by 736
  5. 2023Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Modelscited by 659
  6. 2022GIT: A Generative Image-to-text Transformer for Vision and Languagecited by 536
  7. 2019Attention on Attention for Image Captioningcited by 289
  8. 2019Connecting Vision and Language with Localized Narrativescited by 225
  9. 2021Scaling Up Vision-Language Pre-training for Image Captioningcited by 196
  10. 2015DenseCap: Fully Convolutional Localization Networks for Dense Captioningcited by 176
  11. 2016Deep Interactive Object Selectioncited by 160
  12. 2022GRiT: A Generative Region-to-text Transformer for Object Understandingcited by 119
  13. 2020f-BRS: Rethinking Backpropagating Refinement for Interactive Segmentationcited by 117
  14. 2017DeepIGeoS: A Deep Interactive Geodesic Framework for Medical Image Segmentationcited by 110
  15. 2018Show, Control and Tell: A Framework for Generating Controllable and Grounded Captionscited by 65
  16. 2018Iteratively Trained Interactive Segmentationcited by 52
  17. 2015SentiCap: Generating Image Descriptions with Sentimentscited by 48
  18. 2021Human-like Controllable Image Captioning with Verb-specific Semantic Rolescited by 41
  19. 2021Injecting Semantic Concepts into End-to-End Image Captioningcited by 41
  20. 2016Dense Captioning with Joint Inference and Visual Contextcited by 40
  21. 2020Length-Controllable Image Captioningcited by 36
  22. 2022Comprehending and Ordering Semantics for Image Captioningcited by 33
  23. 2023ConZIC: Controllable Zero-shot Image Captioning by Sampling-Based Polishingcited by 33
  24. 2019Context and Attribute Grounded Dense Captioningcited by 31
  25. 2022Controllable Image Captioning via Promptingcited by 15
  26. 2023Accelerating Vision-Language Pretraining with Free Language Modelingcited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 24; 2024: 34; 2025: 33; 2026: 7.

98 papers.

Sort:

By area:

  1. 2023A Survey on Multimodal Large Language Modelscited by 898
  2. 2023A Comprehensive Overview of Large Language Modelscited by 639
  3. 2023GPT4RoI: Instruction Tuning Large Language Model on Region-of-Interestcited by 300
  4. 2023A Survey on Multimodal Large Language Models for Autonomous Drivingcited by 288
  5. 2023Multimodal Foundation Models: From Specialists to General-Purpose Assistantscited by 247
  6. 2023Woodpecker: Hallucination Correction for Multimodal Large Language Modelscited by 223
  7. 2023Video Understanding with Large Language Models: A Surveycited by 203
  8. 2024IBD: Alleviating Hallucinations in Large Vision-Language Models via Image-Biased Decodingcited by 118
  9. 2023Foundational Models Defining a New Era in Vision: A Survey and Outlookcited by 109
  10. 2025Perception, Reason, Think, and Plan: A Survey on Large Multimodal Reasoning Modelscited by 89
  11. 2025Describe Anything: Detailed Localized Image and Video Captioningcited by 84
  12. 2023Review of Large Vision Models and Visual Prompt Engineeringcited by 77
38 more
  1. 2023Foundational Models in Medical Imaging: A Comprehensive Survey and Future Visioncited by 74
  2. 2024Mitigating Object Hallucinations in Large Vision-Language Models with Assembly of Global and Local Attentioncited by 74
  3. 2023A Comprehensive Survey on Segment Anything Model for Vision and Beyondcited by 64
  4. 2024Reducing Hallucinations in Vision-Language Models via Latent Space Steeringcited by 56
  5. 2025Perceive Anything: Recognize, Explain, Caption, and Segment Anything in Images and Videoscited by 49
  6. 2023A Survey on Segment Anything Model (SAM): Vision Foundation Model Meets Prompt Engineeringcited by 45
  7. 2023FuseCap: Leveraging Large Language Models for Enriched Fused Image Captionscited by 33
  8. 2024ControlMLLM: Training-Free Visual Prompt Learning for Multimodal Large Language Modelscited by 33
  9. 2023M3DBench: Let's Instruct Large Models with Multi-modal 3D Promptscited by 32
  10. 2024PTQ4SAM: Post-Training Quantization for Segment Anythingcited by 32
  11. 2023Tokenize Anything via Promptingcited by 31
  12. 2023Segment and Caption Anythingcited by 30
  13. 2023RegionBLIP: A Unified Multi-modal Pre-training Framework for Holistic and Regional Comprehensioncited by 27
  14. 2024Forging Vision Foundation Models for Autonomous Driving: Challenges, Methodologies, and Opportunitiescited by 24
  15. 2024Attention Prompting on Image for Large Vision-Language Modelscited by 23
  16. 2024It's Never Too Late: Fusing Acoustic Information into Large Language Models for Automatic Speech Recognitioncited by 22
  17. 2025VidCapBench: A Comprehensive Benchmark of Video Captioning for Controllable Text-to-Video Generationcited by 22
  18. 2023Learning the What and How of Annotation in Video Object Segmentationcited by 19
  19. 2024Empowering LLMs with Pseudo-Untrimmed Videos for Audio-Visual Temporal Understandingcited by 18
  20. 2024VidComposition: Can MLLMs Analyze Compositions in Compiled Videos?cited by 18
  21. 2024SAM as the Guide: Mastering Pseudo-Label Refinement in Semi-Supervised Referring Expression Segmentationcited by 17
  22. 2024WTS: A Pedestrian-Centric Traffic Video Dataset for Fine-grained Spatial-Temporal Understandingcited by 17
  23. 2025Generative AI for Cel-Animation: A Surveycited by 17
  24. 2024FINEMATCH: Aspect-based Fine-grained Image and Text Mismatch Detection and Correctioncited by 16
  25. 2024EAGLE: Egocentric AGgregated Language-video Enginecited by 16
  26. 2024On Efficient Variants of Segment Anything Model: A Surveycited by 16
  27. 2025SC-Captioner: Improving Image Captioning with Self-Correction by Reinforcement Learningcited by 16
  28. 2025The Power of Context: How Multimodality Improves Image Super-Resolutioncited by 15
  29. 2024Neural Spectral Decomposition for Dataset Distillationcited by 14
  30. 2024World to Code: Multi-modal Data Generation via Self-Instructed Compositional Captioning and Filteringcited by 14
  31. 2025MIMO: A medical vision language model with visual referring multimodal input and pixel grounding multimodal outputcited by 14
  32. 2024CaRDiff: Video Salient Object Ranking Chain of Thought Reasoning for Saliency Prediction with Diffusioncited by 12
  33. 2023Boosting Segment Anything Model Towards Open-Vocabulary Learningcited by 10
  34. 2024MMRel: Benchmarking Relation Understanding in Multi-Modal Large Language Modelscited by 10
  35. 2024Toward Interactive Regional Understanding in Vision-Large Language Modelscited by 9
  36. 2025CoE: Chain-of-Explanation via Automatic Visual Concept Circuit Description and Polysemanticity Quantificationcited by 9
  37. 2025Caption Anything in Video: Fine-grained Object-centric Captioning via Spatiotemporal Multimodal Promptingcited by 9
  38. 2025Taming Vision-Language Models for Medical Image Analysis: A Comprehensive Reviewcited by 9

50 most cited of 98 All 98 (JSON)