mlsanityData to 14 Oct 2026

ChatGPT Asks, BLIP-2 Answers: Automatic Questioning Towards Enriched Visual Descriptions

arXiv 2303.06594 ↗First posted 12 Mar 2023cs.CV cs.AI cs.LGcited by 86

Not on the map. Why?

Cited by 24 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Asking insightful questions is crucial for acquiring knowledge and expanding our understanding of the world. However, the importance of questioning has been largely overlooked in AI research, where models have been primarily developed to answer questions. With the recent advancements of large language models (LLMs) like ChatGPT, we discover their capability to ask high-quality questions when provided with a suitable prompt. This discovery presents a new opportunity to develop an automatic questioning system. In this paper, we introduce ChatCaptioner, a novel automatic-questioning method deployed in image captioning. Here, ChatGPT is prompted to ask a series of informative questions about images to BLIP-2, a strong vision question-answering model. By keeping acquiring new visual information from BLIP-2's answers, ChatCaptioner is able to generate more enriched image descriptions. We conduct human-subject evaluations on common image caption datasets such as COCO, Conceptual Caption, and WikiArt, and compare ChatCaptioner with BLIP-2 as well as ground truth. Our results demonstrate that ChatCaptioner's captions are significantly more informative, receiving three times as many votes from human evaluators for providing the most image information. Besides, ChatCaptioner identifies 53% more objects within the image than BLIP-2 alone measured by WordNet synset matching. Code is available at https://github.com/Vision-CAIR/ChatCaptioner

Key facts. Cited by 86 papers in the map (data through 14 Oct 2026). Builds on 47 of its 74 references that are in the map. First posted on arXiv 12 Mar 2023.

Builds on

47 of 74 references.

References · 47

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  3. 2022Training language models to follow instructions with human feedbackcited by 15,971
  4. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  5. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  6. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  7. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  8. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  9. 2022Large Language Models are Zero-Shot Reasonerscited by 4,374
  10. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,322
  11. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,148
  12. 2017Deep reinforcement learning from human preferencescited by 3,912
35 more
  1. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
  2. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  3. 2022Scaling Instruction-Finetuned Language Modelscited by 3,024
  4. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,955
  5. 2020Learning to summarize from human feedbackcited by 2,588
  6. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,516
  7. 2022Emergent Abilities of Large Language Modelscited by 2,512
  8. 2015VQA: Visual Question Answeringcited by 2,506
  9. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,046
  10. 2022Large Language Models Are Human-Level Prompt Engineerscited by 1,045
  11. 2022EVA: Exploring the Limits of Masked Visual Representation Learning at Scalecited by 691
  12. 2021Multimodal Few-Shot Learning with Frozen Language Modelscited by 661
  13. 2022Image as a Foreign Language: BEiT Pretraining for All Vision and Vision-Language Taskscited by 511
  14. 2016Visual Dialogcited by 377
  15. 2019Connecting Vision and Language with Localized Narrativescited by 226
  16. 2021VisualGPT: Data-efficient Adaptation of Pretrained Language Models for Image Captioningcited by 191
  17. 2015Large-scale Classification of Fine-Art Paintings: Learning The Right Metric on The Right Featurecited by 182
  18. 2016GuessWhat?! Visual object discovery through multi-modal dialoguecited by 113
  19. 2021ArtEmis: Affective Language for Visual Artcited by 104
  20. 2016Generating Natural Questions About an Imagecited by 56
  21. 2017Visual Question Generation as Dual Task of Visual Question Answeringcited by 41
  22. 2020ERNIE-GEN: An Enhanced Multi-Flow Pre-training and Fine-tuning Framework for Natural Language Generationcited by 40
  23. 2020Asking Questions the Human Way: Scalable Question-Answer Generation from Text Corpuscited by 40
  24. 2017Creativity: Generating Diverse Questions using Variational Autoencoderscited by 24
  25. 2022Question Generation for Reading Comprehension Assessment by Modeling How and What to Askcited by 20
  26. 2016Automatic Generation of Grounded Visual Questionscited by 19
  27. 2019Capturing Greater Context for Question Generationcited by 19
  28. 2022Towards Process-Oriented, Modular, and Versatile Question Generation that Meets Educational Needscited by 19
  29. 2019Learning to Generate Questions by Learning What not to Generatecited by 17
  30. 2021Guiding Visual Question Generationcited by 14
  31. 2019Let's Ask Again: Refine Network for Automatic Question Generationcited by 13
  32. 2018Learning to Caption Images through a Lifetime by Asking Questionscited by 11
  33. 2018Multimodal Differential Network for Visual Question Generationcited by 7
  34. 2020Deep Bayesian Network for Visual Question Generationcited by 7
  35. 2023Inflected Forms Are Redundant in Question Generation Modelscited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 46; 2024: 23; 2025: 12; 2026: 5.

86 papers.

Sort:

By area:

  1. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,711
  2. 2023HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Facecited by 1,190
  3. 2023A Survey on Multimodal Large Language Modelscited by 909
  4. 2023MiniGPT-v2: large language model as a unified interface for vision-language multi-task learningcited by 625
  5. 2023V*: Guided Visual Search as a Core Mechanism in Multimodal LLMscited by 482
  6. 2023An Embodied Generalist Agent in 3D Worldcited by 365
  7. 2023NavGPT: Explicit Reasoning in Vision-and-Language Navigation with Large Language Modelscited by 280
  8. 2023Woodpecker: Hallucination Correction for Multimodal Large Language Modelscited by 226
  9. 2023Physically Grounded Vision-Language Models for Robotic Manipulationcited by 165
  10. 2023Matcher: Segment Anything with One Shot Using All-Purpose Feature Matchingcited by 142
  11. 2023The All-Seeing Project: Towards Panoptic Visual Recognition and Understanding of the Open Worldcited by 116
  12. 2023Foundational Models Defining a New Era in Vision: A Survey and Outlookcited by 110
38 more
  1. 2023Vision-by-Language for Training-Free Compositional Image Retrievalcited by 103
  2. 2023Vision-Language Models in Remote Sensing: Current Progress and Future Trendscited by 93
  3. 2023Improving Multimodal Datasets with Image Captioningcited by 93
  4. 2023VIGC: Visual Instruction Generation and Correctioncited by 80
  5. 2023Intelligent Grimm -- Open-ended Visual Storytelling via Latent Diffusion Modelscited by 77
  6. 2023GPT4Point: A Unified Framework for Point-Language Understanding and Generationcited by 71
  7. 2024OpenBias: Open-set Bias Detection in Text-to-Image Generative Modelscited by 61
  8. 2023DERA: Enhancing Large Language Model Completions with Dialog-Enabled Resolving Agentscited by 52
  9. 2023TIBET: Identifying and Evaluating Biases in Text-to-Image Generative Modelscited by 52
  10. 2023IdealGPT: Iteratively Decomposing Vision and Language Reasoning via Large Language Modelscited by 47
  11. 2024ALOHa: A New Measure for Hallucination in Captioning Modelscited by 45
  12. 2023Exploring Human-Like Translation Strategy with Large Language Modelscited by 44
  13. 2023Video ChatCaptioner: Towards Enriched Spatiotemporal Descriptionscited by 42
  14. 2023Relax Image-Specific Prompt Requirement in SAM: A Single Generic Prompt for Segmenting Camouflaged Objectscited by 40
  15. 2023LLM as A Robotic Brain: Unifying Egocentric Memory and Controlcited by 33
  16. 2023FuseCap: Leveraging Large Language Models for Enriched Fused Image Captionscited by 33
  17. 2024HaloQuest: A Visual Hallucination Dataset for Advancing Multimodal Reasoningcited by 32
  18. 2023HAAR: Text-Conditioned Generative Model of 3D Strand-based Human Hairstylescited by 30
  19. 2024Visual Fact Checker: Enabling High-Fidelity Detailed Caption Generationcited by 28
  20. 2025Explain Before You Answer: A Survey on Compositional Visual Reasoningcited by 28
  21. 2023Chatting Makes Perfect: Chat-based Image Retrievalcited by 25
  22. 2024Democratizing Fine-grained Visual Recognition with Large Language Modelscited by 25
  23. 2023Building an Open-Vocabulary Video CLIP Model with Better Architectures, Optimization and Datacited by 22
  24. 2023Improving Image Captioning Descriptiveness by Ranking and LLM-based Fusioncited by 21
  25. 2023Text Augmented Spatial-aware Zero-shot Referring Image Segmentationcited by 21
  26. 2023MLLMs-Augmented Visual-Language Representation Learningcited by 21
  27. 2025Aligning Vision to Language: Annotation-Free Multimodal Knowledge Graph Construction for Enhanced LLMs Reasoningcited by 19
  28. 2023Bongard-OpenWorld: Few-Shot Reasoning for Free-form Visual Concepts in the Real Worldcited by 18
  29. 2024VLRM: Vision-Language Models act as Reward Models for Image Captioningcited by 18
  30. 2023Rephrase, Augment, Reason: Visual Grounding of Questions for Vision-Language Modelscited by 14
  31. 2023Filling the Image Information Gap for VQA: Prompting Large Language Models to Proactively Ask Questionscited by 14
  32. 2024Towards Truly Zero-shot Compositional Visual Reasoning with LLMs as Programmerscited by 14
  33. 2024USE: Universal Segment Embeddings for Open-Vocabulary Image Segmentationcited by 14
  34. 2023Localized Symbolic Knowledge Distillation for Visual Commonsense Modelscited by 12
  35. 2024HICEScore: A Hierarchical Metric for Image Captioning Evaluationcited by 12
  36. 2023Leveraging VLM-Based Pipelines to Annotate 3D Objectscited by 10
  37. 20243D Question Answering for City Scene Understandingcited by 10
  38. 2024Fine-tuning Multimodal Large Language Models for Product Bundlingcited by 9

50 most cited of 86 All 86 (JSON)