# GENFIG1: Visual Summaries of Scholarly Work as a Challenge for Vision-Language Models

arXiv: https://arxiv.org/abs/2604.04172 · HTML: https://mlsanity.com/paper/2604.04172/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 21 of its 37 references that are in the map.
- First posted on arXiv 5 Apr 2026.

## Placed in

- Multimodal LLMs / vision-language models / [MLLM training and visual instruction tuning](https://mlsanity.com/mllm/mllm-training-and-visual-instruction-tuning/)

## Builds on (most cited first, 21 of 21)

- [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022)
- [Generative Adversarial Networks](https://mlsanity.com/paper/2203.00667/) (2022)
- [GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models](https://mlsanity.com/paper/2112.10741/) (2021)
- [Make-A-Video: Text-to-Video Generation without Text-Video Data](https://mlsanity.com/paper/2209.14792/) (2022)
- [Image Super-Resolution via Iterative Refinement](https://mlsanity.com/paper/2104.07636/) (2021)
- [DVQA: Understanding Data Visualizations via Question Answering](https://mlsanity.com/paper/1801.08163/) (2018)
- [FigureQA: An Annotated Figure Dataset for Visual Reasoning](https://mlsanity.com/paper/1710.07300/) (2017)
- [LayoutDiffusion: Controllable Diffusion Model for Layout-to-image Generation](https://mlsanity.com/paper/2303.17189/) (2023)
- [LayoutDM: Discrete Diffusion Model for Controllable Layout Generation](https://mlsanity.com/paper/2303.08137/) (2023)
- [StarVector: Generating Scalable Vector Graphics Code from Images and Text](https://mlsanity.com/paper/2312.11556/) (2023)
- [AutomaTikZ: Text-Guided Synthesis of Scientific Vector Graphics with TikZ](https://mlsanity.com/paper/2310.00367/) (2023)
- [SciCap: Generating Captions for Scientific Figures](https://mlsanity.com/paper/2110.11624/) (2021)
- [Aligning Large Language Models with Human Preferences through Representation Engineering](https://mlsanity.com/paper/2312.15997/) (2023)
- [SciFIBench: Benchmarking Large Multimodal Models for Scientific Figure Interpretation](https://mlsanity.com/paper/2405.08807/) (2024)
- [OCR-VQGAN: Taming Text-within-Image Generation](https://mlsanity.com/paper/2210.11248/) (2022)
- [SciCap+: A Knowledge Augmented Dataset to Study the Challenges of Scientific Figure Captioning](https://mlsanity.com/paper/2306.03491/) (2023)
- [MMSci: A Dataset for Graduate-Level Multi-Discipline Multimodal Scientific Understanding](https://mlsanity.com/paper/2407.04903/) (2024)
- [AutoFigure: Generating and Refining Publication-Ready Scientific Illustrations](https://mlsanity.com/paper/2602.03828/) (2026)
- [Figure Captioning with Reasoning and Sequence-Level Training](https://mlsanity.com/paper/1906.02850/) (2019)
- [FigGen: Text to Scientific Figure Generation](https://mlsanity.com/paper/2306.00800/) (2023)
- [Sentence-aware Contrastive Learning for Open-Domain Passage Retrieval](https://mlsanity.com/paper/2110.07524/) (2021)

## Built on by (most cited first, 2 of 2)

- [SciForma: Structure-Faithful Generation of Scientific Diagrams](https://mlsanity.com/paper/2607.18091/) (2026)
- [EdiTikZ: Scientific Figure Editing from Revision Trajectories](https://mlsanity.com/paper/2609.01409/) (2026)

## Abstract (arXiv)

In many science papers, "Figure 1" serves as the primary visual summary of the core research idea. These figures are visually simple yet conceptually rich, often requiring significant effort and iteration by human authors to get right, highlighting the difficulty of science visual communication. With this intuition, we introduce GENFIG1, a benchmark for generative AI models (e.g., Vision-Language Models). GENFIG1 evaluates models for their ability to produce figures that clearly express and motivate the central idea of a paper (title, abstract, introduction, and figure caption) as input. Solving GENFIG1 requires more than producing visually appealing graphics: the task entails reasoning for text-to-image generation that couples scientific understanding with visual synthesis. Specifically, models must (i) comprehend and grasp the technical concepts of the paper, (ii) identify the most salient ones, and (iii) design a coherent and aesthetically effective graphic that conveys those concepts visually and is faithful to the input. We curate the benchmark from papers published at top deep-learning conferences, apply stringent quality control, and introduce an automatic evaluation metric that correlates well with expert human judgments. We evaluate a suite of representative models on GENFIG1 and demonstrate that the task presents significant challenges, even for the best-performing systems. We hope this benchmark serves as a foundation for future progress in multimodal AI.