mlsanityData to 7 Oct 2026

MCQG-SRefine: Multiple Choice Question Generation and Evaluation with Iterative Self-Critique, Correction, and Comparison Feedback

arXiv 2410.13191 ↗First posted 17 Oct 2024cs.CL cs.AIcited by 14

Not on the map. Why?

Abstract (arXiv)

Automatic question generation (QG) is essential for AI and NLP, particularly in intelligent tutoring, dialogue systems, and fact verification. Generating multiple-choice questions (MCQG) for professional exams, like the United States Medical Licensing Examination (USMLE), is particularly challenging, requiring domain expertise and complex multi-hop reasoning for high-quality questions. However, current large language models (LLMs) like GPT-4 struggle with professional MCQG due to outdated knowledge, hallucination issues, and prompt sensitivity, resulting in unsatisfactory quality and difficulty. To address these challenges, we propose MCQG-SRefine, an LLM self-refine-based (Critique and Correction) framework for converting medical cases into high-quality USMLE-style questions. By integrating expert-driven prompt engineering with iterative self-critique and self-correction feedback, MCQG-SRefine significantly enhances human expert satisfaction regarding both the quality and difficulty of the questions. Furthermore, we introduce an LLM-as-Judge-based automatic metric to replace the complex and costly expert evaluation process, ensuring reliable and expert-aligned assessments.

Key facts. Cited by 14 papers in the map (data through 7 Oct 2026). Builds on 42 of its 89 references that are in the map. First posted on arXiv 17 Oct 2024.

Builds on

42 of 89 references.

References · 42

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  4. 2023Let's Verify Step by Stepcited by 3,598
  5. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,317
  6. 2022Large Language Models Encode Clinical Knowledgecited by 1,807
  7. 2020What Disease does this Patient Have? A Large-scale Open Domain Question Answering Dataset from Medical Examscited by 1,324
  8. 2023Large Language Models are not Fair Evaluatorscited by 1,050
  9. 2023Teaching Large Language Models to Self-Debugcited by 963
  10. 2023ChatGPT Outperforms Crowd-Workers for Text-Annotation Taskscited by 849
  11. 2023AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedbackcited by 738
  12. 2023Can Large Language Models Be an Alternative to Human Evaluations?cited by 729
30 more
  1. 2023Prometheus: Inducing Fine-grained Evaluation Capability in Language Modelscited by 558
  2. 2021ColBERTv2: Effective and Efficient Retrieval via Lightweight Late Interactioncited by 464
  3. 2024Prometheus 2: An Open Source Language Model Specialized in Evaluating Other Language Modelscited by 431
  4. 2023Large Language Models Are State-of-the-Art Evaluators of Translation Qualitycited by 416
  5. 2023How Far Can Camels Go? Exploring the State of Instruction Tuning on Open Resourcescited by 405
  6. 2023Fine-Grained Human Feedback Gives Better Rewards for Language Model Trainingcited by 399
  7. 2023Understanding the Effects of RLHF on LLM Generalisation and Diversitycited by 386
  8. 2023Evaluating Large Language Models at Evaluating Instruction Followingcited by 290
  9. 2022Generating Sequences by Learning to Self-Correctcited by 271
  10. 2023Automatically Correcting Large Language Models: Surveying the landscape of diverse self-correction strategiescited by 236
  11. 2024AgentClinic: a multimodal agent benchmark to evaluate AI in simulated clinical environmentscited by 215
  12. 2023Towards LLM-based Fact Verification on News Claims with a Hierarchical Step-by-Step Prompting Methodcited by 97
  13. 2023The Generative AI Paradox: "What It Can Create, It May Not Understand"cited by 97
  14. 2023Shepherd: A Critic for Language Model Generationcited by 96
  15. 2024Self-Contrast: Better Reflection Through Inconsistent Solving Perspectivescited by 71
  16. 2023A Comprehensive Analysis of the Effectiveness of Large Language Models as Automatic Dialogue Evaluatorscited by 45
  17. 2021Zero-shot Fact Verification by Claim Generationcited by 42
  18. 2024BioRAG: A RAG-LLM Framework for Biological Question Reasoningcited by 42
  19. 2024JMLR: Joint Medical LLM and Retrieval Training for Enhancing Reasoning and Professional Question Answering Capabilitycited by 32
  20. 2024Constructing Domain-Specific Evaluation Sets for LLM-as-a-judgecited by 28
  21. 2022Educational Question Generation of Children Storybooks via Question Type Distribution Learning and Event-Centric Summarizationcited by 26
  22. 2022StoryER: Automatic Story Evaluation via Ranking, Rating and Reasoningcited by 21
  23. 2023PaniniQA: Enhancing Patient Education Through Interactive Question Answeringcited by 21
  24. 2024Comparing Two Model Designs for Clinical Note Generation; Is an LLM a Useful Evaluator of Consistency?cited by 20
  25. 2023README: Bridging Medical Jargon and Lay Understanding for Patient Education through Data-Centric NLPcited by 17
  26. 2024MedQA-CS: Objective Structured Clinical Examination (OSCE)-Style Benchmark for Evaluating LLM Clinical Skillscited by 16
  27. 2024Towards Leveraging Large Language Models for Automated Medical Q&A Evaluationcited by 14
  28. 2024SYNFAC-EDIT: Synthetic Imitation Edit Feedback for Factual Alignment in Clinical Summarizationcited by 12
  29. 2020Exploring Question-Specific Rewards for Generating Deep Questionscited by 9
  30. 2022QA4QG: Using Question Answering to Constrain Multi-Hop Question Generationcited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 8; 2026: 5.

14 papers.

Sort:

By area:

  1. 2024RARE: Retrieval-Augmented Reasoning Enhancement for Large Language Modelscited by 24
  2. 2025ALFA: Aligning LLMs to Ask Good Questions A Case Study in Clinical Reasoningcited by 24
  3. 2026Knowing When to Abstain: Medical LLMs Under Clinical Uncertaintycited by 13
  4. 2025From Scores to Steps: Diagnosing and Improving LLM Performance in Evidence-Based Medical Calculationscited by 11
  5. 2025DischargeSim: A Simulation Benchmark for Educational Doctor-Patient Communication at Dischargecited by 7
  6. 2025Chatbot To Help Patients Understand Their Healthcited by 4
  7. 2025AttackSeqBench: Benchmarking the Capabilities of LLMs for Attack Sequences Understandingcited by 3
  8. 2025ChatThero: An LLM-Supported Chatbot for Behavior Change and Therapeutic Support in Addiction Recoverycited by 3
  9. 2025Enhancing the Learning Experience: Using Vision-Language Models to Generate Questions for Educational Videoscited by 1
  10. 2025ChatCLIDS: Simulating Persuasive AI Dialogues to Promote Closed-Loop Insulin Adoption in Type 1 Diabetes Carecited by 1
  11. 2026LLM-as-Judge in Education: A Curriculum-Grounded Marking Pipelinecited by 1
  12. 2026Rethinking Patient Education as Multi-turn Multi-modal Interactioncited by 0
2 more
  1. 2026DeepTutor: Towards Agentic Personalized Tutoringcited by 0
  2. 2026AI-accelerated End-to-End Framework for Rapid Professional Upskillingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025It is Too Many Options: Pitfalls of Multiple-Choice Questions in Generative AI and Medical Education
  2. 2026Cognitively Diverse Multiple-Choice Question Generation: A Hybrid Multi-Agent Framework with Large Language Models
  3. 2026Orchestrating LLM Agents for Scientific Research: A Pilot Study of Multiple Choice Question (MCQ) Generation and Evaluation
  4. 2026KNIGHT: Knowledge Graph-Driven Multiple-Choice Question Generation with Adaptive Hardness Calibration
  5. 2026Automated Benchmark Generation from Domain Guidelines Informed by Bloom's Taxonomy
  6. 2025Generate-Then-Validate: A Novel Question Generation Approach Using Small Language Models
  7. 2026Can Large Language Models Self-Correct in Medical Question Answering? An Exploratory Study
  8. 2025Benchmarking Critical Questions Generation: A Challenging Reasoning Task for Large Language Models