# PosterSum: A Multimodal Benchmark for Scientific Poster Summarization

arXiv: https://arxiv.org/abs/2502.17540 · HTML: https://mlsanity.com/paper/2502.17540/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Placed in Chart understanding and anomaly detection, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 32 of its 53 references that are in the map.
- First posted on arXiv 24 Feb 2025.

## Placed in

- Multimodal LLMs / vision-language models / [Chart understanding and anomaly detection](https://mlsanity.com/mllm/chart-understanding-and-anomaly-detection/)

## Builds on (most cited first, 30 of 32)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [Improved Baselines with Visual Instruction Tuning](https://mlsanity.com/paper/2310.03744/) (2023)
- [CLIPScore: A Reference-free Evaluation Metric for Image Captioning](https://mlsanity.com/paper/2104.08718/) (2021)
- [MMBench: Is Your Multi-modal Model an All-around Player?](https://mlsanity.com/paper/2307.06281/) (2023)
- [MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI](https://mlsanity.com/paper/2311.16502/) (2023)
- [A Call for Clarity in Reporting BLEU Scores](https://mlsanity.com/paper/1804.08771/) (2018)
- [MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts](https://mlsanity.com/paper/2310.02255/) (2023)
- [ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoning](https://mlsanity.com/paper/2203.10244/) (2022)
- [FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation](https://mlsanity.com/paper/2305.14251/) (2023)
- [SummaC: Re-Visiting NLI-based Models for Inconsistency Detection in Summarization](https://mlsanity.com/paper/2111.09525/) (2021)
- [QAFactEval: Improved QA-Based Factual Consistency Evaluation for Summarization](https://mlsanity.com/paper/2112.08542/) (2021)
- [SlideVQA: A Dataset for Document Visual Question Answering on Multiple Images](https://mlsanity.com/paper/2301.04883/) (2023)
- [TLDR: Extreme Summarization of Scientific Documents](https://mlsanity.com/paper/2004.15011/) (2020)
- [DocLLM: A layout-aware generative language model for multimodal document understanding](https://mlsanity.com/paper/2401.00908/) (2023)
- [mPLUG-DocOwl2: High-resolution Compressing for OCR-free Multi-page Document Understanding](https://mlsanity.com/paper/2409.03420/) (2024)
- [LayoutLLM: Layout Instruction Tuning with Large Language Models for Document Understanding](https://mlsanity.com/paper/2404.05225/) (2024)
- [CapsFusion: Rethinking Image-Text Data at Scale](https://mlsanity.com/paper/2310.20550/) (2023)
- [SPIQA: A Dataset for Multimodal Question Answering on Scientific Papers](https://mlsanity.com/paper/2407.09413/) (2024)
- [ScisummNet: A Large Annotated Corpus and Content-Impact Models for Scientific Paper Summarization with Citation Networks](https://mlsanity.com/paper/1909.01716/) (2019)
- [ERNIE-Layout: Layout Knowledge Enhanced Pre-training for Visually-rich Document Understanding](https://mlsanity.com/paper/2210.06155/) (2022)
- [DocFormerv2: Local Features for Document Understanding](https://mlsanity.com/paper/2306.01733/) (2023)
- [MMOCR: A Comprehensive Toolbox for Text Detection, Recognition and Understanding](https://mlsanity.com/paper/2108.06543/) (2021)
- [TalkSumm: A Dataset and Scalable Annotation Method for Scientific Paper Summarization Based on Conference Talks](https://mlsanity.com/paper/1906.01351/) (2019)
- [Mix and Reason: Reasoning over Semantic Topology with Data Mixing for Domain Generalization](https://mlsanity.com/paper/2210.07571/) (2022)
- [ACLSum: A New Dataset for Aspect-based Summarization of Scientific Publications](https://mlsanity.com/paper/2403.05303/) (2024)
- [Leveraging Information Bottleneck for Scientific Document Summarization](https://mlsanity.com/paper/2110.01280/) (2021)
- [SciNews: From Scholarly Complexities to Public Narratives -- A Dataset for Scientific News Report Generation](https://mlsanity.com/paper/2403.17768/) (2024)
- [Select and Summarize: Scene Saliency for Movie Script Summarization](https://mlsanity.com/paper/2404.03561/) (2024)
- [TSTR: Too Short to Represent, Summarize with Details! Intro-Guided Extended Summary Generation](https://mlsanity.com/paper/2206.00847/) (2022)

## Built on by (most cited first, 10 of 10)

- [Paper2Poster: Towards Multimodal Poster Automation from Scientific Papers](https://mlsanity.com/paper/2505.21497/) (2025)
- [P2P: Automated Paper-to-Poster Generation and Fine-Grained Benchmark](https://mlsanity.com/paper/2505.17104/) (2025)
- [PosterGen: Aesthetic-Aware Multi-Modal Paper-to-Poster Generation via Multi-Agent LLMs](https://mlsanity.com/paper/2508.17188/) (2025)
- [PosterForest: Hierarchical Multi-Agent Collaboration for Scientific Poster Generation](https://mlsanity.com/paper/2508.21720/) (2025)
- [Any2Poster: Any-Source Poster Generation Across Modalities and Domains](https://mlsanity.com/paper/2606.02915/) (2026)
- [PRISMM-Bench: A Benchmark of Peer-Review Grounded Multimodal Inconsistencies](https://mlsanity.com/paper/2510.16505/) (2025)
- [SciPostGen: Bridging the Gap between Scientific Papers and Poster Layouts](https://mlsanity.com/paper/2511.22490/) (2025)
- [Training Multi-Image Vision Agents via End2End Reinforcement Learning](https://mlsanity.com/paper/2512.08980/) (2025)
- [VideoAgent: Personalized Synthesis of Scientific Videos](https://mlsanity.com/paper/2509.11253/) (2025)
- [TextRich: A Multi-Domain Benchmark for Detecting AI-Generated Text-Rich Images from GPT-Image-2](https://mlsanity.com/paper/2606.19259/) (2026)

## Abstract (arXiv)

Generating accurate and concise textual summaries from multimodal documents is challenging, especially when dealing with visually complex content like scientific posters. We introduce PosterSum, a novel benchmark to advance the development of vision-language models that can understand and summarize scientific posters into research paper abstracts. Our dataset contains 16,305 conference posters paired with their corresponding abstracts as summaries. Each poster is provided in image format and presents diverse visual understanding challenges, such as complex layouts, dense text regions, tables, and figures. We benchmark state-of-the-art Multimodal Large Language Models (MLLMs) on PosterSum and demonstrate that they struggle to accurately interpret and summarize scientific posters. We propose Segment & Summarize, a hierarchical method that outperforms current MLLMs on automated metrics, achieving a 3.14% gain in ROUGE-L. This will serve as a starting point for future research on poster summarization.