# QGEval: Benchmarking Multi-dimensional Evaluation for Question Generation

arXiv: https://arxiv.org/abs/2406.05707 · HTML: https://mlsanity.com/paper/2406.05707/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 41 references that are in the map.
- First posted on arXiv 9 Jun 2024.

## Builds on (most cited first, 28 of 28)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [SQuAD: 100,000+ Questions for Machine Comprehension of Text](https://mlsanity.com/paper/1606.05250/) (2016)
- [HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering](https://mlsanity.com/paper/1809.09600/) (2018)
- [G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment](https://mlsanity.com/paper/2303.16634/) (2023)
- [Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?](https://mlsanity.com/paper/2202.12837/) (2022)
- [BLEURT: Learning Robust Metrics for Text Generation](https://mlsanity.com/paper/2004.04696/) (2020)
- [BARTScore: Evaluating Generated Text as Text Generation](https://mlsanity.com/paper/2106.11520/) (2021)
- [Large Language Models Are State-of-the-Art Evaluators of Translation Quality](https://mlsanity.com/paper/2302.14520/) (2023)
- [MoverScore: Text Generation Evaluating with Contextualized Embeddings and Earth Mover Distance](https://mlsanity.com/paper/1909.02622/) (2019)
- [Towards a Unified Multi-Dimensional Evaluator for Text Generation](https://mlsanity.com/paper/2210.07197/) (2022)
- [A Survey of Evaluation Metrics Used for NLG Systems](https://mlsanity.com/paper/2008.12009/) (2020)
- [Neural Question Generation from Text: A Preliminary Study](https://mlsanity.com/paper/1704.01792/) (2017)
- [Towards a Better Metric for Evaluating Question Generation Systems](https://mlsanity.com/paper/1808.10192/) (2018)
- [Guiding the Growth: Difficulty-Controllable Question Generation through Step-by-Step Rewriting](https://mlsanity.com/paper/2105.11698/) (2021)
- [Generative Language Models for Paragraph-Level Question Generation](https://mlsanity.com/paper/2210.03992/) (2022)
- [Varifocal Question Generation for Fact-checking](https://mlsanity.com/paper/2210.12400/) (2022)
- [Improving Unsupervised Question Answering via Summarization-Informed Question Generation](https://mlsanity.com/paper/2109.07954/) (2021)
- [Question Generation for Reading Comprehension Assessment by Modeling How and What to Ask](https://mlsanity.com/paper/2204.02908/) (2022)
- [RQUGE: Reference-Free Metric for Evaluating Question Generation by Answering the Question](https://mlsanity.com/paper/2211.01482/) (2022)
- [Guiding Visual Question Generation](https://mlsanity.com/paper/2110.08226/) (2021)
- [Quiz Design Task: Helping Teachers Create Quizzes with Automated Question Generation](https://mlsanity.com/paper/2205.01730/) (2022)
- [Diversify Question Generation with Retrieval-Augmented Style Transfer](https://mlsanity.com/paper/2310.14503/) (2023)
- [QRelScore: Better Evaluating Generated Questions with Deeper Understanding of Context-aware Relevance](https://mlsanity.com/paper/2204.13921/) (2022)
- [QAScore -- An Unsupervised Unreferenced Metric for the Question Generation Evaluation](https://mlsanity.com/paper/2210.04320/) (2022)
- [Evaluation of Question Generation Needs More References](https://mlsanity.com/paper/2305.16626/) (2023)

## Built on by (most cited first, 17 of 17)

- [AskQE: Question Answering as Automatic Evaluation for Machine Translation](https://mlsanity.com/paper/2504.11582/) (2025)
- [KRETA: A Benchmark for Korean Reading and Reasoning in Text-Rich VQA Attuned to Diverse Visual Contexts](https://mlsanity.com/paper/2508.19944/) (2025)
- [Factuality Beyond Coherence: Evaluating LLM Watermarking Methods for Medical Texts](https://mlsanity.com/paper/2509.07755/) (2025)
- [Multi-Agent Collaborative Framework For Math Problem Generation](https://mlsanity.com/paper/2511.03958/) (2025)
- [Persona-SQ: A Personalized Suggested Question Generation Framework For Real-world Documents](https://mlsanity.com/paper/2412.12445/) (2024)
- [A Comparative Study of Competency Question Elicitation Methods from Ontology Requirements](https://mlsanity.com/paper/2507.02989/) (2025)
- [Leveraging Graph Retrieval-Augmented Generation to Support Learners' Understanding of Knowledge Concepts in MOOCs](https://mlsanity.com/paper/2505.10074/) (2025)
- [HopWeaver: Cross-Document Synthesis of High-Quality and Authentic Multi-Hop Questions](https://mlsanity.com/paper/2505.15087/) (2025)
- [EssayBench: Evaluating Large Language Models in Multi-Genre Chinese Essay Writing](https://mlsanity.com/paper/2506.02596/) (2025)
- [How Significant Are the Real Performance Gains? An Unbiased Evaluation Framework for GraphRAG](https://mlsanity.com/paper/2506.06331/) (2025)
- [Improving Cross-Format Robustness in Language Models with Multi-Format Training](https://mlsanity.com/paper/2606.11643/) (2026)
- [ErrEval: Error-Aware Evaluation for Question Generation through Explicit Diagnostics](https://mlsanity.com/paper/2601.10406/) (2026)
- [Beyond Factual QA: Mentorship-Oriented Question Answering over Long-Form Multilingual Content](https://mlsanity.com/paper/2601.17173/) (2026)
- [RARE: Redundancy-Aware Retrieval Evaluation Framework for High-Similarity Corpora](https://mlsanity.com/paper/2604.19047/) (2026)
- [Query Generation Pipeline with Enhanced Answerability Assessment for Financial Information Retrieval](https://mlsanity.com/paper/2511.05000/) (2025)
- [You Really Didn't Get That? Benchmarking Social Pragmatic Inference for Indirect and Playful Chinese Online Comments](https://mlsanity.com/paper/2609.04384/) (2026)
- [ChemCLIR-Bench: Benchmarking Cross-Lingual Information Retrieval in Multilingual Chemical Patents](https://mlsanity.com/paper/2609.23231/) (2026)

## Abstract (arXiv)

Automatically generated questions often suffer from problems such as unclear expression or factual inaccuracies, requiring a reliable and comprehensive evaluation of their quality. Human evaluation is widely used in the field of question generation (QG) and serves as the gold standard for automatic metrics. However, there is a lack of unified human evaluation criteria, which hampers consistent and reliable evaluations of both QG models and automatic metrics. To address this, we propose QGEval, a multi-dimensional Evaluation benchmark for Question Generation, which evaluates both generated questions and existing automatic metrics across 7 dimensions: fluency, clarity, conciseness, relevance, consistency, answerability, and answer consistency. We demonstrate the appropriateness of these dimensions by examining their correlations and distinctions. Through consistent evaluations of QG models and automatic metrics with QGEval, we find that 1) most QG models perform unsatisfactorily in terms of answerability and answer consistency, and 2) existing metrics fail to align well with human judgments when evaluating generated questions across the 7 dimensions. We expect this work to foster the development of both QG technologies and their evaluation.