# LaMP-Cap: Personalized Figure Caption Generation With Multimodal Figure Profiles

arXiv: https://arxiv.org/abs/2506.06561 · HTML: https://mlsanity.com/paper/2506.06561/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 44 references that are in the map.
- First posted on arXiv 6 Jun 2025.

## Builds on (most cited first, 25 of 25)

- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization](https://mlsanity.com/paper/1912.08777/) (2019)
- [LaMP: When Large Language Models Meet Personalization](https://mlsanity.com/paper/2304.11406/) (2023)
- [Chart-to-Text: A Large-Scale Benchmark for Chart Summarization](https://mlsanity.com/paper/2203.06486/) (2022)
- [UniChart: A Universal Vision-language Pretrained Model for Chart Comprehension and Reasoning](https://mlsanity.com/paper/2305.14761/) (2023)
- [MatCha: Enhancing Visual Language Pretraining with Math Reasoning and Chart Derendering](https://mlsanity.com/paper/2212.09662/) (2022)
- [VisText: A Benchmark for Semantically Rich Chart Captioning](https://mlsanity.com/paper/2307.05356/) (2023)
- [Personalization of Large Language Models: A Survey](https://mlsanity.com/paper/2411.00027/) (2024)
- [LongLaMP: A Benchmark for Personalized Long-form Text Generation](https://mlsanity.com/paper/2407.11016/) (2024)
- [Democratizing Large Language Models via Personalized Parameter-Efficient Fine-tuning](https://mlsanity.com/paper/2402.04401/) (2024)
- [ChartAssisstant: A Universal Chart Multimodal Language Model via Chart-to-Table Pre-training and Multitask Instruction Tuning](https://mlsanity.com/paper/2401.02384/) (2024)
- [Chart-to-Text: Generating Natural Language Descriptions for Charts by Adapting the Transformer Model](https://mlsanity.com/paper/2010.09142/) (2020)
- [SciCap: Generating Captions for Scientific Figures](https://mlsanity.com/paper/2110.11624/) (2021)
- [Engaging Image Captioning Via Personality](https://mlsanity.com/paper/1810.10665/) (2018)
- [Personalized Large Language Models](https://mlsanity.com/paper/2402.09269/) (2024)
- [ExPerT: Effective and Explainable Evaluation of Personalized Long-Form Text Generation](https://mlsanity.com/paper/2501.14956/) (2025)
- [SciCap+: A Knowledge Augmented Dataset to Study the Challenges of Scientific Figure Captioning](https://mlsanity.com/paper/2306.03491/) (2023)
- [Striking a Balance: Reader Takeaways and Preferences when Integrating Text and Charts](https://mlsanity.com/paper/2208.01780/) (2022)
- [Summaries as Captions: Generating Figure Captions for Scientific Documents with Automated Text Summarization](https://mlsanity.com/paper/2302.12324/) (2023)
- [Towards Understanding How Readers Integrate Charts and Captions: A Case Study with Line Charts](https://mlsanity.com/paper/2101.08235/) (2021)
- [Multi-LLM Collaborative Caption Generation in Scientific Documents](https://mlsanity.com/paper/2501.02552/) (2025)
- [SciCapenter: Supporting Caption Composition for Scientific Figures with Machine-Generated Captions and Ratings](https://mlsanity.com/paper/2403.17784/) (2024)
- [Understanding How Paper Writers Use AI-Generated Captions in Figure Caption Writing](https://mlsanity.com/paper/2501.06317/) (2025)
- [Do Large Multimodal Models Solve Caption Generation for Scientific Figures? Lessons Learned from SciCap Challenge 2023](https://mlsanity.com/paper/2501.19353/) (2025)
- [RONA: Pragmatically Diverse Image Captioning with Coherence Relations](https://mlsanity.com/paper/2503.10997/) (2025)

## Built on by (most cited first, 2 of 2)

- [Autonomous Research Agents: A Survey of AI Scientists and the Verification Gap](https://mlsanity.com/paper/2608.05179/) (2026)
- [Leveraging Author-Specific Context for Scientific Figure Caption Generation: 3rd SciCap Challenge](https://mlsanity.com/paper/2510.07993/) (2025)

## Abstract (arXiv)

Figure captions are crucial for helping readers understand and remember a figure's key message. Many models have been developed to generate these captions, helping authors compose better quality captions more easily. Yet, authors almost always need to revise generic AI-generated captions to match their writing style and the domain's style, highlighting the need for personalization. Despite language models' personalization (LaMP) advances, these technologies often focus on text-only settings and rarely address scenarios where both inputs and profiles are multimodal. This paper introduces LaMP-Cap, a dataset for personalized figure caption generation with multimodal figure profiles. For each target figure, LaMP-Cap provides not only the needed inputs, such as figure images, but also up to three other figures from the same document--each with its image, caption, and figure-mentioning paragraphs--as a profile to characterize the context. Experiments with four LLMs show that using profile information consistently helps generate captions closer to the original author-written ones. Ablation studies reveal that images in the profile are more helpful than figure-mentioning paragraphs, highlighting the advantage of using multimodal profiles over text-only ones.