# Personalizing Multimodal Large Language Models for Image Captioning: An Experimental Analysis

arXiv: https://arxiv.org/abs/2412.03665 · HTML: https://mlsanity.com/paper/2412.03665/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 27 of its 70 references that are in the map.
- First posted on arXiv 4 Dec 2024.

## Placed in

- Multimodal LLMs / vision-language models / [VLM benchmarks and visual reasoning](https://mlsanity.com/mllm/vlm-benchmarks-and-visual-reasoning/)

## Builds on (most cited first, 27 of 27)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models](https://mlsanity.com/paper/2301.12597/) (2023)
- [Flamingo: a Visual Language Model for Few-Shot Learning](https://mlsanity.com/paper/2204.14198/) (2022)
- [Prefix-Tuning: Optimizing Continuous Prompts for Generation](https://mlsanity.com/paper/2101.00190/) (2021)
- [QLoRA: Efficient Finetuning of Quantized LLMs](https://mlsanity.com/paper/2305.14314/) (2023)
- [Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond](https://mlsanity.com/paper/2308.12966/) (2023)
- [mPLUG-Owl: Modularization Empowers Large Language Models with Multimodality](https://mlsanity.com/paper/2304.14178/) (2023)
- [Gemini: A Family of Highly Capable Multimodal Models](https://mlsanity.com/paper/2312.11805/) (2023)
- [Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMs](https://mlsanity.com/paper/2406.16860/) (2024)
- [DoRA: Weight-Decomposed Low-Rank Adaptation](https://mlsanity.com/paper/2402.09353/) (2024)
- [GPT Understands, Too](https://mlsanity.com/paper/2103.10385/) (2021)
- [CogVLM: Visual Expert for Pretrained Language Models](https://mlsanity.com/paper/2311.03079/) (2023)
- [MiniGPT-v2: large language model as a unified interface for vision-language multi-task learning](https://mlsanity.com/paper/2310.09478/) (2023)
- [LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Model](https://mlsanity.com/paper/2304.15010/) (2023)
- [OpenFlamingo: An Open-Source Framework for Training Large Autoregressive Vision-Language Models](https://mlsanity.com/paper/2308.01390/) (2023)
- [ClipCap: CLIP Prefix for Image Captioning](https://mlsanity.com/paper/2111.09734/) (2021)
- [SPHINX: The Joint Mixing of Weights, Tasks, and Visual Embeddings for Multi-modal Large Language Models](https://mlsanity.com/paper/2311.07575/) (2023)
- [Woodpecker: Hallucination Correction for Multimodal Large Language Models](https://mlsanity.com/paper/2310.16045/) (2023)
- [WARP: Word-level Adversarial ReProgramming](https://mlsanity.com/paper/2101.00121/) (2021)
- [SVIT: Scaling up Visual Instruction Tuning](https://mlsanity.com/paper/2307.04087/) (2023)
- [Revisiting Image Captioning Training Paradigm via Direct CLIP-based Optimization](https://mlsanity.com/paper/2408.14547/) (2024)

## Built on by (most cited first, 23 of 23)

- [OThink-MR1: Stimulating multimodal generalized reasoning capabilities via dynamic reinforcement learning](https://mlsanity.com/paper/2503.16081/) (2025)
- [pySpatial: Generating 3D Visual Programs for Zero-Shot Spatial Reasoning](https://mlsanity.com/paper/2603.00905/) (2026)
- [CCCaption: Dual-Reward Reinforcement Learning for Complete and Correct Image Captioning](https://mlsanity.com/paper/2602.21655/) (2026)
- [See or Recall: A Sanity Check for the Role of Vision in Solving Visualization Question Answer Tasks with Multimodal LLMs](https://mlsanity.com/paper/2504.09809/) (2025)
- [R-Genie: Reasoning-Guided Generative Image Editing](https://mlsanity.com/paper/2505.17768/) (2025)
- [Do MLLMs Really Understand Space? A Mathematical Reasoning Evaluation](https://mlsanity.com/paper/2602.11635/) (2026)
- [VF-Eval: Evaluating Multimodal LLMs for Generating Feedback on AIGC Videos](https://mlsanity.com/paper/2505.23693/) (2025)
- [CrossGuard: Safeguarding MLLMs against Joint-Modal Implicit Malicious Attacks](https://mlsanity.com/paper/2510.17687/) (2025)
- [Retrieving Counterfactuals Improves Visual In-Context Learning](https://mlsanity.com/paper/2603.16737/) (2026)
- [Multifaceted Evaluation of Audio-Visual Capability for MLLMs: Effectiveness, Efficiency, Generalizability and Robustness](https://mlsanity.com/paper/2504.16936/) (2025)
- [Image Corruption-Inspired Membership Inference Attacks against Large Vision-Language Models](https://mlsanity.com/paper/2506.12340/) (2025)
- [How Multimodal Large Language Models Support Access to Visual Information: A Diary Study With Blind and Low Vision People](https://mlsanity.com/paper/2602.13469/) (2026)
- [What Changed? Detecting and Evaluating Instruction-Guided Image Edits with Multimodal Large Language Models](https://mlsanity.com/paper/2505.20405/) (2025)
- [Diffusion Language Models: An Experimental Analysis](https://mlsanity.com/paper/2606.19475/) (2026)
- [MIDAS: Multi-Image Dispersion and Semantic Reconstruction for Jailbreaking MLLMs](https://mlsanity.com/paper/2603.00565/) (2026)
- [Granulon: Awakening Pixel-Level Visual Encoders with Adaptive Multi-Granularity Semantics for MLLM](https://mlsanity.com/paper/2603.08800/) (2026)
- [Spotlight and Shadow: Attention-Guided Dual-Anchor Introspective Decoding for MLLM Hallucination Mitigation](https://mlsanity.com/paper/2604.10071/) (2026)
- [VisuCraft: Enhancing Large Vision-Language Models for Complex Visual-Guided Creative Content Generation via Structured Information Extraction](https://mlsanity.com/paper/2508.02890/) (2025)
- [OIG-Bench: A Multi-Agent Annotated Benchmark for Multimodal One-Image Guides Understanding](https://mlsanity.com/paper/2510.00069/) (2025)
- [Covert Visual Prompt Injection against Commercial Multimodal Large Language Models](https://mlsanity.com/paper/2603.29418/) (2026)
- [Jailbreaking Multimodal Large Language Models using Multi-Clip Video](https://mlsanity.com/paper/2606.02111/) (2026)
- [Generative Retrieval for Unsupervised Text-Based Person Search](https://mlsanity.com/paper/2609.12965/) (2026)
- [DocAttriBench: Benchmarking Answer Grounding in Document Visual Question Answering](https://mlsanity.com/paper/2609.20574/) (2026)

## Abstract (arXiv)

The task of image captioning demands an algorithm to generate natural language descriptions of visual inputs. Recent advancements have seen a convergence between image captioning research and the development of Large Language Models (LLMs) and Multimodal LLMs -- like GPT-4V and Gemini -- which extend the capabilities of text-only LLMs to multiple modalities. This paper investigates whether Multimodal LLMs can supplant traditional image captioning networks by evaluating their performance on various image description benchmarks. We explore both the zero-shot capabilities of these models and their adaptability to different semantic domains through fine-tuning methods, including prompt learning, prefix tuning, and low-rank adaptation. Our results demonstrate that while Multimodal LLMs achieve impressive zero-shot performance, fine-tuning for specific domains while maintaining their generalization capabilities intact remains challenging. We discuss the implications of these findings for future research in image captioning and the development of more adaptable Multimodal LLMs.