# Diagram-Driven Course Questions Generation

arXiv: https://arxiv.org/abs/2411.17771 · HTML: https://mlsanity.com/paper/2411.17771/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 42 references that are in the map.
- First posted on arXiv 26 Nov 2024.

## Builds on (most cited first, 18 of 18)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI](https://mlsanity.com/paper/2311.16502/) (2023)
- [CIDEr: Consensus-based Image Description Evaluation](https://mlsanity.com/paper/1411.5726/) (2014)
- [Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering](https://mlsanity.com/paper/2209.09513/) (2022)
- [Multimodal Chain-of-Thought Reasoning in Language Models](https://mlsanity.com/paper/2302.00923/) (2023)
- [IconQA: A New Benchmark for Abstract Diagram Understanding and Visual Language Reasoning](https://mlsanity.com/paper/2110.13214/) (2021)
- [M3CoT: A Novel Benchmark for Multi-Domain Multi-step Multi-modal Chain-of-Thought](https://mlsanity.com/paper/2405.16473/) (2024)
- [FLEUR: An Explainable Reference-Free Evaluation Metric for Image Captioning Using a Large Multimodal Model](https://mlsanity.com/paper/2406.06004/) (2024)
- [Information Maximizing Visual Question Generation](https://mlsanity.com/paper/1903.11207/) (2019)
- [EvoChart: A Benchmark and a Self-Training Approach Towards Real-World Chart Understanding](https://mlsanity.com/paper/2409.01577/) (2024)
- [Learning by Asking Questions](https://mlsanity.com/paper/1712.01238/) (2017)
- [ISAAQ -- Mastering Textbook Questions with Pre-trained Transformers and Bottom-Up and Top-Down Attention](https://mlsanity.com/paper/2010.00562/) (2020)
- [K-VQG: Knowledge-aware Visual Question Generation for Common-sense Acquisition](https://mlsanity.com/paper/2203.07890/) (2022)
- [VProChart: Answering Chart Question through Visual Perception Alignment Agent and Programmatic Solution Reasoning](https://mlsanity.com/paper/2409.01667/) (2024)
- [Visual Question Generation for Class Acquisition of Unknown Objects](https://mlsanity.com/paper/1808.01821/) (2018)
- [ConVQG: Contrastive Visual Question Generation with Multimodal Guidance](https://mlsanity.com/paper/2402.12846/) (2024)

## Built on by (most cited first, 2 of 2)

- [Graph-to-Vision: Multi-graph Understanding and Reasoning using Vision-Language Models](https://mlsanity.com/paper/2503.21435/) (2025)
- [MAPS: Multi-Agent Personality Shaping for Collaborative Reasoning](https://mlsanity.com/paper/2503.16905/) (2025)

## Abstract (arXiv)

Visual Question Generation (VQG) research focuses predominantly on natural images while neglecting the diagram, which is a critical component in educational materials. To meet the needs of pedagogical assessment, we propose the Diagram-Driven Course Questions Generation (DDCQG) task and construct DiagramQG, a comprehensive dataset with 15,720 diagrams and 25,798 questions across 37 subjects and 371 courses. Our approach employs course and input text constraints to generate course-relevant questions about specific diagram elements. We reveal three challenges of DDCQG: domain-specific knowledge requirements across courses, long-tail distribution in course coverage, and high information density in diagrams. To address these, we propose the Hierarchical Knowledge Integration framework (HKI-DDCQG), which utilizes trainable CLIP for identifying relevant diagram patches, leverages frozen vision-language models for knowledge extraction, and generates questions with trainable T5. Experiments demonstrate that HKI-DDCQG outperforms existing models on DiagramQG while maintaining strong generalizability across natural image datasets, establishing a strong baseline for DDCQG.