# CODE-GEN: A Human-in-the-Loop RAG-Based Agentic AI System for Multiple-Choice Question Generation

arXiv: https://arxiv.org/abs/2604.03926 · HTML: https://mlsanity.com/paper/2604.03926/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in Agentic RAG, agents and memory, one of 15 directions in Retrieval-augmented generation.
- Builds on 16 of its 32 references that are in the map.
- First posted on arXiv 5 Apr 2026.

## Placed in

- Retrieval-augmented generation / [Agentic RAG, agents and memory](https://mlsanity.com/rag/agentic-rag-agents-and-memory/)
- LLM agents and tool use / [Educational agents and math problem solving](https://mlsanity.com/agents/educational-agents-and-math-problem-solving/)

## Builds on (most cited first, 16 of 16)

- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [Billion-scale similarity search with GPUs](https://mlsanity.com/paper/1702.08734/) (2017)
- [The Landscape of Emerging AI Agent Architectures for Reasoning, Planning, and Tool Calling: A Survey](https://mlsanity.com/paper/2404.11584/) (2024)
- [LLM-based Agents Suffer from Hallucinations: A Survey of Taxonomy, Methods, and Directions](https://mlsanity.com/paper/2509.18970/) (2025)
- [A Comparative Study of AI-Generated (GPT-4) and Human-crafted MCQs in Programming Education](https://mlsanity.com/paper/2312.03173/) (2023)
- [Automated Educational Question Generation at Different Bloom's Skill Levels using Large Language Models: Strategies and Evaluation](https://mlsanity.com/paper/2408.04394/) (2024)
- [Evaluating the Effectiveness of LLMs in Introductory Computer Science Education: A Semester-Long Field Study](https://mlsanity.com/paper/2404.13414/) (2024)
- [MIRAGE-Bench: LLM Agent is Hallucinating and Where to Find Them](https://mlsanity.com/paper/2507.21017/) (2025)
- [A Multi-AI Agent System for Autonomous Optimization of Agentic AI Solutions via Iterative Refinement and LLM-Driven Feedback Loops](https://mlsanity.com/paper/2412.17149/) (2024)
- [Evolution of AI in Education: Agentic Workflows](https://mlsanity.com/paper/2504.20082/) (2025)
- [Improving Assessment of Tutoring Practices using Retrieval-Augmented Generation](https://mlsanity.com/paper/2402.14594/) (2024)
- [Lessons from Building StackSpot AI: A Contextualized AI Coding Assistant](https://mlsanity.com/paper/2311.18450/) (2023)
- [Generating AI Literacy MCQs: A Multi-Agent LLM Approach](https://mlsanity.com/paper/2412.00970/) (2024)
- [From First Draft to Final Insight: A Multi-Agent Approach for Feedback Generation](https://mlsanity.com/paper/2505.04869/) (2025)
- [Facilitating Instructors-LLM Collaboration for Problem Design in Introductory Programming Classrooms](https://mlsanity.com/paper/2504.01259/) (2025)
- [Evaluating adaptive and generative AI-based feedback and recommendations in a knowledge-graph-integrated programming learning system](https://mlsanity.com/paper/2603.24940/) (2026)

## Built on by (most cited first, 1 of 1)

- [TeachMateGPT: A Multi-Agent Knowledge-Grounded Framework for Pedagogical Assessment Generation from Science Curriculum Materials](https://mlsanity.com/paper/2608.13708/) (2026)

## Abstract (arXiv)

We present CODE-GEN, a human-in-the-Loop, retrieval-augmented generation (RAG)-based agentic AI system for generating context-aligned multiple-choice questions to develop student code reasoning and comprehension abilities. CODE-GEN employs an agentic AI architecture in which a Generator agent produces multiple-choice coding comprehension questions aligned with course-specific learning objectives, while a Validator agent independently assesses content quality across seven pedagogical dimensions. Both agents are augmented with specialized tools that enhance computational accuracy and verify code outputs. To evaluate the effectiveness of CODE-GEN, we conducted an evaluation study involving six human subject-matter experts (SMEs) who judged 288 AI-generated questions. The SMEs produced a total of 2,016 human-AI rating pairs, indicating agreement or disagreement with the assessments of Validator, along with 131 instances of qualitative feedback. Analyses of SME judgments show strong system performance, with human-validated success rates ranging from 79.9% to 98.6% across the seven pedagogical dimensions. The analysis of qualitative feedback reveals that CODE-GEN achieves high reliability on dimensions well suited to computational verification and explicit criteria matching, including question clarity, code validity, concept alignment, and correct answer validity. In contrast, human expertise remains essential for dimensions requiring deeper instructional judgment, such as designing pedagogically meaningful distractors and providing high-quality feedback that reinforces understanding. These findings inform the strategic allocation of human and AI effort in AI-assisted educational content generation.