Multimodal QUD: Inquisitive Questions from Scientific Figures
Not on the map. Why?
Abstract (arXiv)
Discourse comprehension in complex documents often involves continuously posing and resolving Questions Under Discussion (QUDs). While QUD frameworks have so far focused on text, scientific literature is inherently multimodal: figures convey discourse goals distinct from their textual counterparts, thus invoking implicit questions that the surrounding text answers. In scientific discovery, knowing the right questions to ask is as important as knowing how to answer them, yet this capability remains largely absent in current models. In this work, we extend QUD to multimodal discourse in scientific literature, targeting questions evoked by figures that are (1) inquisitive, i.e., not resolved in the prior context; (2) salient, i.e., relevant to the paper's research claims and addressed later in the paper; (3) grounded in visual insights. To benchmark model capability to generate such questions, we introduce MQUD, a dataset of 1,250 figure-evoked questions from 56 scientific papers, including 708 questions annotated by the original authors of these papers. Our experiments show that open-source VLMs such as Qwen 3.5 predominantly ask questions that can be answered by the figure alone. Fine-tuning on MQUD teaches the model to ask scientifically relevant, inquisitive questions that target the role a figure plays in the paper's argument.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 60 of its 99 references that are in the map. First posted on arXiv 26 Apr 2026.
Builds on
60 of 99 references.
References · 60
- 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoningcited by 1,479
- 2024The AI Scientist: Towards Fully Automated Open-Ended Scientific Discoverycited by 1,071
- 2019PlotQA: Reasoning over Scientific Plotscited by 321
- 2017FigureQA: An Annotated Figure Dataset for Visual Reasoningcited by 319
- 2023SciBench: Evaluating College-Level Scientific Problem-Solving Abilities of Large Language Modelscited by 256
- 2024Multimodal ArXiv: A Dataset for Improving Scientific Comprehension of Large Vision-Language Modelscited by 139
- 2025SAEs Are Good for Steering -- If You Select the Right Featurescited by 78
- 2024SPIQA: A Dataset for Multimodal Question Answering on Scientific Paperscited by 75
- 2023SciGraphQA: A Large-Scale Synthetic Multi-Turn Question-Answering Dataset for Scientific Graphscited by 68
- 2021IsoScore: Measuring the Uniformity of Embedding Space Utilizationcited by 53
- 2024SciFIBench: Benchmarking Large Multimodal Models for Scientific Figure Interpretationcited by 48
- 2021Controllable Open-ended Question Generation with A New Question Type Ontologycited by 36
38 more
- 2020Inquisitive Question Generation for High Level Text Comprehensioncited by 32
- 2023ReFACT: Updating Text-to-Image Models by Editing the Text Encodercited by 32
- 2025ChartMuseum: Testing Visual Reasoning Capabilities of Large Vision-Language Modelscited by 32
- 2025Pixels Versus Priors: Controlling Knowledge Priors in Vision-Language Models through Visual Counterfactscited by 32
- 2024What Do VLMs NOTICE? A Mechanistic Interpretability Pipeline for Gaussian-Noise-free Text-Image Corruption and Evaluationcited by 30
- 2025MedBrowseComp: Benchmarking Medical Deep Research and Computer Usecited by 30
- 2023The Best of Both Worlds: Accurate Global and Personalized Models through Federated Learning with Data-Free Hyper-Knowledge Distillationcited by 25
- 2025Enhancing Retrieval-Augmented Generation: A Study of Best Practicescited by 23
- 2025Can Language Models Learn Typologically Implausible Languages?cited by 23
- 2025PeerQA: A Scientific Question Answering Dataset from Peer Reviewscited by 22
- 2021Discourse Comprehension: A Question Answering Framework to Represent Sentence Connectionscited by 19
- 2023QUDEVAL: The Evaluation of Questions Under Discussion Discourse Parsingcited by 19
- 2025SpaceGNN: Multi-Space Graph Neural Network for Node Anomaly Detection with Extremely Limited Labelscited by 17
- 2025Forgotten Polygons: Multimodal Large Language Models are Shape-Blindcited by 16
- 2016Viziometrics: Analyzing Visual Information in the Scientific Literaturecited by 15
- 2022COMPS: Conceptual Minimal Pair Sentences for testing Robust Property Knowledge and its Inheritance in Pre-trained Language Modelscited by 15
- 2024Cross-Care: Assessing the Healthcare Implications of Pre-training Data on Language Model Biascited by 15
- 2023Elaborative Simplification as Implicit Questions Under Discussioncited by 14
- 2022Discourse Analysis via Questions and Answers: Parsing Dependency Structures of Questions Under Discussioncited by 13
- 2025QUDsim: Quantifying Discourse Similarities in LLM-Generated Textcited by 13
- 2023Stable Anisotropic Regularizationcited by 12
- 2023Mixed-Precision Quantization for Federated Learning on Resource-Constrained Heterogeneous Devicescited by 11
- 2024cPAPERS: A Dataset of Situated and Multimodal Interactive Conversations in Scientific Paperscited by 11
- 2024Characterizing the Role of Similarity in the Property Inferences of Language Modelscited by 11
- 2025CRISP: Persistent Concept Unlearning via Sparse Autoencoderscited by 11
- 2019Decomposing Generalization: Models of Generic, Habitual, and Episodic Statementscited by 10
- 2025Can Multimodal Foundation Models Understand Schematic Diagrams? An Empirical Study on Information-Seeking QA over Scientific Paperscited by 10
- 2023Multilingual Code Co-Evolution Using Large Language Modelscited by 9
- 2025The Harmonic Structure of Information Contourscited by 9
- 2025OKBench: Democratizing LLM Evaluation with Fully Automated, On-Demand, Open Knowledge Benchmarkingcited by 9
- 2024Which questions should I answer? Salience Prediction of Inquisitive Questionscited by 8
- 2020Help! Need Advice on Identifying Advicecited by 6
- 2023One-Versus-Others Attention: Scalable Multimodal Integration for Biomedical Datacited by 6
- 2024Making FETCH! Happen: Finding Emergent Dog Whistles Through Common Habitatscited by 6
- 2025Vision-and-Language Training Helps Deploy Taxonomic Knowledge but Does Not Fundamentally Alter Itcited by 6
- 2022How people talk about each other: Modeling Generalized Intergroup Bias and Emotioncited by 5
- 2024Recovering Labels from Local Updates in Federated Learningcited by 5
- 2024exLong: Generating Exceptional Behavior Tests with Large Language Modelscited by 4
50 most cited of 60 All 60 (JSON)
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026Do you "see" what I "see"? A Multi-panel Visual Question Answer Dataset for Large Language Model Chart Analysis
- 2025Enhancing Scientific Visual Question Answering through Multimodal Reasoning and Ensemble Modeling
- 2026Signal or Noise? Modality Contribution and Cooperation in Multimodal GraphRAG ↗
- 2025MicroVQA: A Multimodal Reasoning Benchmark for Microscopy-Based Scientific Research
- 2025A Simple Data Augmentation Strategy for Text-in-Image Scientific VQA
- 2025MAVIS: A Benchmark for Multimodal Source Attribution in Long-form Visual Question Answering
- 2025VeriSciQA: An Auto-Verified Dataset for Scientific Visual Question Answering
- 2025Chain of Questions: Guiding Multimodal Curiosity in Language Models