mlsanityData to 14 Oct 2026

FuseCap: Leveraging Large Language Models for Enriched Fused Image Captions

arXiv 2305.17718 ↗First posted 28 May 2023cs.CV cs.AI cs.CLcited by 33

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

The advent of vision-language pre-training techniques enhanced substantial progress in the development of models for image captioning. However, these models frequently produce generic captions and may omit semantically important image details. This limitation can be traced back to the image-text datasets; while their captions typically offer a general description of image content, they frequently omit salient details. Considering the magnitude of these datasets, manual reannotation is impractical, emphasizing the need for an automated approach. To address this challenge, we leverage existing captions and explore augmenting them with visual details using "frozen" vision experts including an object detector, an attribute recognizer, and an Optical Character Recognizer (OCR). Our proposed method, FuseCap, fuses the outputs of such vision experts with the original captions using a large language model (LLM), yielding comprehensive image descriptions. We automatically curate a training set of 12M image-enriched caption pairs. These pairs undergo extensive evaluation through both quantitative and qualitative analyses. Subsequently, this data is utilized to train a captioning generation BLIP-based model. This model outperforms current state-of-the-art approaches, producing more precise and detailed descriptions, demonstrating the effectiveness of the proposed data-centric approach. We release this large-scale dataset of enriched image-caption pairs for the community.

Key facts. Cited by 33 papers in the map (data through 14 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 58 of its 82 references that are in the map. First posted on arXiv 28 May 2023.

Builds on

58 of 82 references.

Closest ancestors · 5

Same direction, newest first

  1. 2022Position-guided Text Prompt for Vision-Language Pre-training19 Dec
  2. 2022VLP: A Survey on Vision-Language Pre-training18 Feb
  3. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation28 Jan
  4. 2021Scaling Up Vision-Language Pre-training for Image Captioning24 Nov
  5. 2021SimVLM: Simple Visual Language Model Pretraining with Weak Supervision24 Aug

Other references · 53

  1. 2017Attention Is All You Needcited by 60,361
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  5. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  6. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  7. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,639
  8. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  9. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  10. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,182
  11. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,864
  12. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
38 more
  1. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  2. 2022Scaling Instruction-Finetuned Language Modelscited by 3,024
  3. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,656
  4. 2022InstructPix2Pix: Learning to Follow Image Editing Instructionscited by 2,611
  5. 2016Aggregated Residual Transformations for Deep Neural Networkscited by 2,546
  6. 2021CLIPScore: A Reference-free Evaluation Metric for Image Captioningcited by 2,424
  7. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,046
  8. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,619
  9. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,553
  10. 2015Show, Attend and Tell: Neural Image Caption Generation with Visual Attentioncited by 1,423
  11. 2017Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answeringcited by 1,312
  12. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,296
  13. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,216
  14. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,214
  15. 2014Deep Visual-Semantic Alignments for Generating Image Descriptionscited by 968
  16. 2019VL-BERT: Pre-training of Generic Visual-Linguistic Representationscited by 954
  17. 2017Revisiting Unreasonable Effectiveness of Data in Deep Learning Eracited by 919
  18. 2014Show and Tell: A Neural Image Caption Generatorcited by 879
  19. 2022PaLI: A Jointly-Scaled Multilingual Language-Image Modelcited by 766
  20. 2022OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Frameworkcited by 745
  21. 2023Instruction Tuning with GPT-4cited by 686
  22. 2022GIT: A Generative Image-to-text Transformer for Vision and Languagecited by 537
  23. 2014Long-term Recurrent Convolutional Networks for Visual Recognition and Descriptioncited by 484
  24. 2018nocaps: novel object captioning at scalecited by 444
  25. 2021Unifying Vision-and-Language Tasks via Text Generationcited by 412
  26. 2019Scene Text Visual Question Answeringcited by 331
  27. 2019Character Region Awareness for Text Detectioncited by 278
  28. 2021An Empirical Study of Training End-to-End Vision-and-Language Transformerscited by 271
  29. 2023Data-centric Artificial Intelligence: A Surveycited by 203
  30. 2015DenseCap: Fully Convolutional Localization Networks for Dense Captioningcited by 176
  31. 2022Scene Text Recognition with Permuted Autoregressive Sequence Modelscited by 149
  32. 2014Deep Fragment Embeddings for Bidirectional Image Sentence Mappingcited by 99
  33. 2023Caption Anything: Interactive Image Description with Diverse Multimodal Controlscited by 99
  34. 2023ChatGPT Asks, BLIP-2 Answers: Automatic Questioning Towards Enriched Visual Descriptionscited by 86
  35. 2018Discriminability objective for training descriptive captionscited by 67
  36. 2023Data-centric AI: Perspectives and Challengescited by 43
  37. 2016Dense Captioning with Joint Inference and Visual Contextcited by 40
  38. 2018Show, Tell and Discriminate: Image Captioning by Self-retrieval with Partially Labeled Datacited by 37

50 most cited of 53 All 53 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 2; 2024: 14; 2025: 9; 2026: 8.

33 papers.

Sort:

By area:

  1. 2024Procedure-Aware Surgical Video-language Pretraining with Hierarchical Knowledge Augmentationcited by 39
  2. 2025CapRL: Stimulating Dense Image Caption Capabilities via Reinforcement Learningcited by 38
  3. 2024Question Aware Vision Transformer for Multimodal Reasoningcited by 32
  4. 2024Pathways on the Image Manifold: Image Editing via Video Generationcited by 27
  5. 2023Improving Image Captioning Descriptiveness by Ranking and LLM-based Fusioncited by 21
  6. 2024Enhancing Visual Question Answering through Question-Driven Image Captions as Promptscited by 18
  7. 2024Positive-Augmented Contrastive Learning for Vision-and-Language Evaluation and Trainingcited by 16
  8. 2025HQ-CLIP: Leveraging Large Vision-Language Models to Create High-Quality Image-Text Datasets and CLIP Modelscited by 11
  9. 2024Context-aware Difference Distilling for Multi-change Captioningcited by 10
  10. 2025MobileUse: A GUI Agent with Hierarchical Reflection for Autonomous Mobile Operationcited by 9
  11. 2024From Descriptive Richness to Bias: Unveiling the Dark Side of Generative Image Caption Enrichmentcited by 8
  12. 2026CapRL++: Unified Reinforcement Learning with Verifiable Rewards for Dense Image and Video Captioningcited by 8
21 more
  1. 2024AgriBench: A Hierarchical Agriculture Benchmark for Multimodal Large Language Modelscited by 7
  2. 2025GAIA: A Global, Multi-modal, Multi-scale Vision-Language Dataset for Remote Sensing Image Analysiscited by 7
  3. 2025Direct Visual Grounding by Directing Attention of Visual Tokenscited by 7
  4. 2023Establishing Performance Baselines in Fine-Tuning, Retrieval-Augmented Generation and Soft-Prompting for Non-Specialist LLM Userscited by 6
  5. 2024The Pitfalls of Defining Hallucinationcited by 5
  6. 2024FocusChat: Text-guided Long Video Understanding via Spatiotemporal Information Filteringcited by 5
  7. 2024Generative AI for Immersive Communication: The Next Frontier in Internet-of-Senses Through 6Gcited by 4
  8. 2025MM-GEN: Enhancing Task Performance Through Targeted Multimodal Data Curationcited by 4
  9. 2025VL4Gaze: Unleashing Vision-Language Models for Gaze Followingcited by 3
  10. 2025Caption Injection for Optimization in Generative Search Enginecited by 2
  11. 2024Paint Outside the Box: Synthesizing and Selecting Training Data for Visual Groundingcited by 1
  12. 2026See and Fix the Flaws: Enabling VLMs and Diffusion Models to Comprehend Visual Artifacts via Agentic Data Synthesiscited by 1
  13. 2026SafeCap: Improving LVLM Safety with Image Captioning Reinforcement Learningcited by 1
  14. 2024Survey: Transformer-based Models in Data Modality Conversioncited by 0
  15. 2024Patchfinder: Leveraging Visual Language Models for Accurate Information Retrieval using Model Uncertaintycited by 0
  16. 2025From Satellite to Street: A Hybrid Framework Integrating Stable Diffusion and PanoGAN for Consistent Cross-View Synthesiscited by 0
  17. 2026Cross-modal Identity Mapping: Minimizing Information Loss in Modality Conversion via Reinforcement Learningcited by 0
  18. 2026ITIScore: An Image-to-Text-to-Image Rating Framework for the Image Captioning Ability of MLLMscited by 0
  19. 2026Beyond Single Plots: A Benchmark for Question Answering on Multi-Chartscited by 0
  20. 2026Test-Time Hallucination Control in Large Vision-Language Modelscited by 0
  21. 2026G-CARL: Grounded Checklist-Aligned Reward Learning for Patient-Oriented Medical Report Interpretationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023A Picture is Worth More Than 77 Text Tokens: Evaluating CLIP-Style Models on Dense Captions
  2. 2024DreamLIP: Language-Image Pre-training with Long Captions
  3. 2023VeCLIP: Improving CLIP Training via Visual-enriched Captions
  4. 2024BLIP3-KALE: Knowledge Augmented Large-Scale Dense Captions
  5. 2024Bridging the Visual Gap: Fine-Tuning Multimodal Models with Knowledge-Adapted Captions
  6. 2024Benchmarking and Improving Detail Image Caption
  7. 2025A Multimodal Recaptioning Framework to Account for Perceptual Diversity Across Languages in Vision-Language Modeling
  8. 2024Personalizing Multimodal Large Language Models for Image Captioning: An Experimental Analysis