# MIRROR: A Novel Approach for the Automated Evaluation of Open-Ended Question Generation

arXiv: https://arxiv.org/abs/2410.12893 · HTML: https://mlsanity.com/paper/2410.12893/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 45 references that are in the map.
- First posted on arXiv 16 Oct 2024.

## Builds on (most cited first, 25 of 25)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Self-Refine: Iterative Refinement with Self-Feedback](https://mlsanity.com/paper/2303.17651/) (2023)
- [Crowdsourcing Multiple Choice Science Questions](https://mlsanity.com/paper/1707.06209/) (2017)
- [Can Large Language Models Be an Alternative to Human Evaluations?](https://mlsanity.com/paper/2305.01937/) (2023)
- [RRHF: Rank Responses to Align Language Models with Human Feedback without tears](https://mlsanity.com/paper/2304.05302/) (2023)
- [Fine-Grained Human Feedback Gives Better Rewards for Language Model Training](https://mlsanity.com/paper/2306.01693/) (2023)
- [Re-evaluating Evaluation in Text Summarization](https://mlsanity.com/paper/2010.07100/) (2020)
- [RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMs](https://mlsanity.com/paper/2404.08555/) (2024)
- [Towards a Better Metric for Evaluating Question Generation Systems](https://mlsanity.com/paper/1808.10192/) (2018)
- [How Ready are Pre-trained Abstractive Models and LLMs for Legal Case Judgement Summarization?](https://mlsanity.com/paper/2306.01248/) (2023)
- [Generative Language Models for Paragraph-Level Question Generation](https://mlsanity.com/paper/2210.03992/) (2022)
- [Applicability of Large Language Models and Generative Models for Legal Case Judgement Summarization](https://mlsanity.com/paper/2407.12848/) (2024)
- [A Novel Multi-Stage Prompting Approach for Language Agnostic MCQ Generation using GPT](https://mlsanity.com/paper/2401.07098/) (2024)
- [Exploring the Capabilities of Prompted Large Language Models in Educational and Assessment Applications](https://mlsanity.com/paper/2405.11579/) (2024)
- [How Ready Are Generative Pre-trained Large Language Models for Explaining Bengali Grammatical Errors?](https://mlsanity.com/paper/2406.00039/) (2024)
- [Fact-based Court Judgment Prediction](https://mlsanity.com/paper/2311.13350/) (2023)
- [Nonet at SemEval-2023 Task 6: Methodologies for Legal Evaluation](https://mlsanity.com/paper/2310.11049/) (2023)
- [Harnessing the Power of Prompt-based Techniques for Generating School-Level Questions using Large Language Models](https://mlsanity.com/paper/2312.01032/) (2023)
- [How Effective is GPT-4 Turbo in Generating School-Level Questions from Textbooks Based on Bloom's Revised Taxonomy?](https://mlsanity.com/paper/2406.15211/) (2024)
- [Questioning Biases in Case Judgment Summaries: Legal Datasets or Large Language Models?](https://mlsanity.com/paper/2312.00554/) (2023)
- [Artificial Intelligence (AI) in Legal Data Mining](https://mlsanity.com/paper/2405.14707/) (2024)
- [Prompted Zero-Shot Multi-label Classification of Factual Incorrectness in Machine-Generated Summaries](https://mlsanity.com/paper/2312.01087/) (2023)
- [Multi-Label Classification of COVID-Tweets Using Large Language Models](https://mlsanity.com/paper/2312.10748/) (2023)

## Built on by (most cited first, 6 of 6)

- [Opportunities and Challenges of LLMs in Education: An NLP Perspective](https://mlsanity.com/paper/2507.22753/) (2025)
- [Prompt Engineering Using GPT for Word-Level Code-Mixed Language Identification in Low-Resource Dravidian Languages](https://mlsanity.com/paper/2411.04025/) (2024)
- [Reflecting in the Reflection: Integrating a Socratic Questioning Framework into Automated AI-Based Question Generation](https://mlsanity.com/paper/2601.14798/) (2026)
- [Cancer-Answer: Empowering Cancer Care with Advanced Large Language Models](https://mlsanity.com/paper/2411.06946/) (2024)
- [CryptoLLM: Unleashing the Power of Prompted LLMs for SmartQnA and Classification of Crypto Posts](https://mlsanity.com/paper/2411.07917/) (2024)
- [The Impact of Item-Writing Flaws on Difficulty and Discrimination in Item Response Theory](https://mlsanity.com/paper/2503.10533/) (2025)

## Abstract (arXiv)

Automatic question generation is a critical task that involves evaluating question quality by considering factors such as engagement, pedagogical value, and the ability to stimulate critical thinking. These aspects require human-like understanding and judgment, which automated systems currently lack. However, human evaluations are costly and impractical for large-scale samples of generated questions. Therefore, we propose a novel system, MIRROR (Multi-LLM Iterative Review and Response for Optimized Rating), which leverages large language models (LLMs) to automate the evaluation process for questions generated by automated question generation systems. We experimented with several state-of-the-art LLMs, such as GPT-4, Gemini, and Llama2-70b. We observed that the scores of human evaluation metrics, namely relevance, appropriateness, novelty, complexity, and grammaticality, improved when using the feedback-based approach called MIRROR, tending to be closer to the human baseline scores. Furthermore, we observed that Pearson's correlation coefficient between GPT-4 and human experts improved when using our proposed feedback-based approach, MIRROR, compared to direct prompting for evaluation. Error analysis shows that our proposed approach, MIRROR, significantly helps to improve relevance and appropriateness.