mlsanityData to 14 Oct 2026

Visual Clues: Bridging Vision and Language Foundations for Image Paragraph Captioning

arXiv 2206.01843 ↗First posted 3 Jun 2022cs.CV cs.AI cs.CLcited by 21

Not on the map. Why?

Abstract (arXiv)

People say, "A picture is worth a thousand words". Then how can we get the rich information out of the image? We argue that by using visual clues to bridge large pretrained vision foundation models and language models, we can do so without any extra cross-modal training. Thanks to the strong zero-shot capability of foundation models, we start by constructing a rich semantic representation of the image (e.g., image tags, object attributes / locations, captions) as a structured textual prompt, called visual clues, using a vision foundation model. Based on visual clues, we use large language model to produce a series of comprehensive descriptions for the visual content, which is then verified by the vision model again to select the candidate that aligns best with the image. We evaluate the quality of generated descriptions by quantitative and qualitative measurement. The results demonstrate the effectiveness of such a structured semantic representation.

Key facts. Cited by 21 papers in the map (data through 14 Oct 2026). Builds on 41 of its 58 references that are in the map. First posted on arXiv 3 Jun 2022.

Builds on

41 of 58 references.

References · 41

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  4. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,322
  5. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,148
  6. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,864
  7. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,656
  8. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,516
  9. 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,181
  10. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,046
  11. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,462
  12. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,274
29 more
  1. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,121
  2. 2020On Faithfulness and Factuality in Abstractive Summarizationcited by 900
  3. 2016SPICE: Semantic Propositional Image Caption Evaluationcited by 824
  4. 2021Florence: A New Foundation Model for Computer Visioncited by 804
  5. 2022OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Frameworkcited by 745
  6. 2017DailyDialog: A Manually Labelled Multi-turn Dialogue Datasetcited by 652
  7. 2021FLAVA: A Foundational Language And Vision Alignment Modelcited by 649
  8. 2021SimVLM: Simple Visual Language Model Pretraining with Weak Supervisioncited by 647
  9. 2019Object Detection in 20 Years: A Surveycited by 531
  10. 2022Winoground: Probing Vision and Language Models for Visio-Linguistic Compositionalitycited by 529
  11. 2022Socratic Models: Composing Zero-Shot Multimodal Reasoning with Languagecited by 513
  12. 2019Unified Vision-Language Pre-Training for Image Captioning and VQAcited by 494
  13. 2021Unifying Vision-and-Language Tasks via Text Generationcited by 412
  14. 2021An Empirical Study of GPT-3 for Few-Shot Knowledge-Based VQAcited by 301
  15. 2021VisualGPT: Data-efficient Adaptation of Pretrained Language Models for Image Captioningcited by 191
  16. 2021Dynamic Head: Unifying Object Detection Heads with Attentionscited by 187
  17. 2016Visual Storytellingcited by 176
  18. 2020Detecting Hallucinated Content in Conditional Neural Sequence Generationcited by 152
  19. 2021Uni-Perceiver: Pre-training Unified Architecture for Generic Perception for Zero-shot and Few-shot Taskscited by 125
  20. 2016A Hierarchical Approach for Generating Descriptive Image Paragraphscited by 104
  21. 2022Language Models Can See: Plugging Visual Controls in Text Generationcited by 90
  22. 2017Towards Diverse and Natural Image Descriptions via a Conditional GANcited by 59
  23. 2018Inverse Cooking: Recipe Generation from Food Imagescited by 47
  24. 2017Recurrent Topic-Transition GAN for Visual Paragraph Generationcited by 25
  25. 2018Beyond Narrative Description: Generating Poetry from Images by Multi-Adversarial Trainingcited by 17
  26. 2018Diverse and Coherent Paragraph Generation from Imagescited by 13
  27. 2019Curiosity-driven Reinforcement Learning for Diverse Visual Paragraph Generationcited by 10
  28. 2019Convolutional Auto-encoding of Sentence Topics for Image Paragraph Generationcited by 8
  29. 2021Matching Visual Features to Hierarchical Semantic Topics for Image Paragraph Captioningcited by 2

Built on by

Cited by year

Citing papers per year: 2022: 4; 2023: 10; 2024: 6; 2025: 1.

21 papers.

Sort:

By area:

  1. 2022Linearly Mapping from Image to Text Spacecited by 142
  2. 2022Vision-Language Pre-training: Basics, Recent Advances, and Future Trendscited by 117
  3. 2022PromptCap: Prompt-Guided Task-Aware Image Captioningcited by 104
  4. 2023MM-VID: Advancing Video Understanding with GPT-4V(ision)cited by 83
  5. 2023Learning Concise and Descriptive Attributes for Visual Recognitioncited by 73
  6. 2023Towards Language Models That Can See: Computer Vision Through the LENS of Natural Languagecited by 53
  7. 2023ASSISTGUI: Task-Oriented Desktop Graphical User Interface Automationcited by 45
  8. 2024Visual Fact Checker: Enabling High-Fidelity Detailed Caption Generationcited by 28
  9. 2023IC3: Image Captioning by Committee Consensuscited by 20
  10. 2023Rephrase, Augment, Reason: Visual Grounding of Questions for Vision-Language Modelscited by 14
  11. 2024HICEScore: A Hierarchical Metric for Image Captioning Evaluationcited by 12
  12. 2024LOC-ZSON: Language-driven Object-Centric Zero-Shot Object Retrieval and Navigationcited by 8
9 more
  1. 2022Do DALL-E and Flamingo Understand Each Other?cited by 7
  2. 2023Album Storytelling with Iterative Story-aware Captioning and Large Language Modelscited by 7
  3. 2023InfoVisDial: An Informative Visual Dialogue Dataset by Bridging Large Multimodal and Language Modelscited by 5
  4. 2025Semantic Document Derendering: SVG Reconstruction via Vision-Language Modelingcited by 4
  5. 2024MGIMM: Multi-Granularity Instruction Multimodal Model for Attribute-Guided Remote Sensing Image Detailed Descriptioncited by 3
  6. 2024Surveying the Landscape of Image Captioning Evaluation: A Comprehensive Taxonomy, Trends and Metrics Analysiscited by 1
  7. 2023Exploring Annotation-free Image Captioning with Retrieval-augmented Pseudo Sentence Generationcited by 0
  8. 2023Apollo: Zero-shot MultiModal Reasoning with Multiple Expertscited by 0
  9. 2024Simplifying Multimodality: Unimodal Approach to Multimodal Challenges in Radiology with General-Domain Large Language Modelcited by 0