mlsanityData to 7 Oct 2026

Personalizing Multimodal Large Language Models for Image Captioning: An Experimental Analysis

arXiv 2412.03665 ↗First posted 4 Dec 2024cs.CV cs.AI cs.CLcited by 23

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

The task of image captioning demands an algorithm to generate natural language descriptions of visual inputs. Recent advancements have seen a convergence between image captioning research and the development of Large Language Models (LLMs) and Multimodal LLMs -- like GPT-4V and Gemini -- which extend the capabilities of text-only LLMs to multiple modalities. This paper investigates whether Multimodal LLMs can supplant traditional image captioning networks by evaluating their performance on various image description benchmarks. We explore both the zero-shot capabilities of these models and their adaptability to different semantic domains through fine-tuning methods, including prompt learning, prefix tuning, and low-rank adaptation. Our results demonstrate that while Multimodal LLMs achieve impressive zero-shot performance, fine-tuning for specific domains while maintaining their generalization capabilities intact remains challenging. We discuss the implications of these findings for future research in image captioning and the development of more adaptable Multimodal LLMs.

Key facts. Cited by 23 papers in the map (data through 7 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 27 of its 70 references that are in the map. First posted on arXiv 4 Dec 2024.

Builds on

27 of 70 references.

Closest ancestors · 3

Same direction, newest first

  1. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond24 Aug
  2. 2023OpenFlamingo: An Open-Source Framework for Training Large Autoregressive Vision-Language Models2 Aug
  3. 2022Flamingo: a Visual Language Model for Few-Shot Learning29 Apr

Other references · 24

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2023GPT-4 Technical Reportcited by 18,132
  4. 2022Training language models to follow instructions with human feedbackcited by 15,712
  5. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  6. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  7. 2023Visual Instruction Tuningcited by 7,792
  8. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  9. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  10. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  11. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,038
  12. 2024Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMscited by 822
12 more
  1. 2023Gemini: A Family of Highly Capable Multimodal Modelscited by 810
  2. 2024DoRA: Weight-Decomposed Low-Rank Adaptationcited by 740
  3. 2021GPT Understands, Toocited by 715
  4. 2023CogVLM: Visual Expert for Pretrained Language Modelscited by 704
  5. 2023MiniGPT-v2: large language model as a unified interface for vision-language multi-task learningcited by 617
  6. 2023LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Modelcited by 602
  7. 2021ClipCap: CLIP Prefix for Image Captioningcited by 557
  8. 2023SPHINX: The Joint Mixing of Weights, Tasks, and Visual Embeddings for Multi-modal Large Language Modelscited by 283
  9. 2023Woodpecker: Hallucination Correction for Multimodal Large Language Modelscited by 223
  10. 2021WARP: Word-level Adversarial ReProgrammingcited by 220
  11. 2023SVIT: Scaling up Visual Instruction Tuningcited by 161
  12. 2024Revisiting Image Captioning Training Paradigm via Direct CLIP-based Optimizationcited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 10; 2026: 13.

23 papers.

Sort:

By area:

  1. 2025OThink-MR1: Stimulating multimodal generalized reasoning capabilities via dynamic reinforcement learningcited by 24
  2. 2026pySpatial: Generating 3D Visual Programs for Zero-Shot Spatial Reasoningcited by 12
  3. 2025See or Recall: A Sanity Check for the Role of Vision in Solving Visualization Question Answer Tasks with Multimodal LLMscited by 7
  4. 2026CCCaption: Dual-Reward Reinforcement Learning for Complete and Correct Image Captioningcited by 7
  5. 2025R-Genie: Reasoning-Guided Generative Image Editingcited by 6
  6. 2025VF-Eval: Evaluating Multimodal LLMs for Generating Feedback on AIGC Videoscited by 4
  7. 2025CrossGuard: Safeguarding MLLMs against Joint-Modal Implicit Malicious Attackscited by 4
  8. 2025Multifaceted Evaluation of Audio-Visual Capability for MLLMs: Effectiveness, Efficiency, Generalizability and Robustnesscited by 3
  9. 2025Image Corruption-Inspired Membership Inference Attacks against Large Vision-Language Modelscited by 3
  10. 2026Do MLLMs Really Understand Space? A Mathematical Reasoning Evaluationcited by 3
  11. 2026How Multimodal Large Language Models Support Access to Visual Information: A Diary Study With Blind and Low Vision Peoplecited by 3
  12. 2026Retrieving Counterfactuals Improves Visual In-Context Learningcited by 3
11 more
  1. 2025What Changed? Detecting and Evaluating Instruction-Guided Image Edits with Multimodal Large Language Modelscited by 2
  2. 2026Diffusion Language Models: An Experimental Analysiscited by 2
  3. 2026MIDAS: Multi-Image Dispersion and Semantic Reconstruction for Jailbreaking MLLMscited by 1
  4. 2026Granulon: Awakening Pixel-Level Visual Encoders with Adaptive Multi-Granularity Semantics for MLLMcited by 1
  5. 2026Spotlight and Shadow: Attention-Guided Dual-Anchor Introspective Decoding for MLLM Hallucination Mitigationcited by 1
  6. 2025VisuCraft: Enhancing Large Vision-Language Models for Complex Visual-Guided Creative Content Generation via Structured Information Extractioncited by 0
  7. 2025OIG-Bench: A Multi-Agent Annotated Benchmark for Multimodal One-Image Guides Understandingcited by 0
  8. 2026Covert Visual Prompt Injection against Commercial Multimodal Large Language Modelscited by 0
  9. 2026Jailbreaking Multimodal Large Language Models using Multi-Clip Videocited by 0
  10. 2026Generative Retrieval for Unsupervised Text-Based Person Searchcited by 0
  11. 2026DocAttriBench: Benchmarking Answer Grounding in Document Visual Question Answeringcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025MM-SeR: Multimodal Self-Refinement for Lightweight Image Captioning
  2. 2023CapsFusion: Rethinking Image-Text Data at Scale
  3. 2023On the Performance of Multimodal Language Models
  4. 2026Multi-Modal LLM based Image Captioning in ICT: Bridging the Gap Between General and Industry Domain
  5. 2023CLAMP: Contrastive LAnguage Model Prompt-tuning
  6. 2023A Survey of Vision-Language Pre-training from the Lens of Multimodal Machine Translation
  7. 2025Unveiling Effective In-Context Configurations for Image Captioning: An External & Internal Analysis
  8. 2025A Multimodal Recaptioning Framework to Account for Perceptual Diversity Across Languages in Vision-Language Modeling