mlsanityData to 14 Oct 2026

Orchestrating LLM Agents for Scientific Research: A Pilot Study of Multiple Choice Question (MCQ) Generation and Evaluation

arXiv 2602.18891 ↗First posted 21 Feb 2026cs.CY cs.AI cs.HCcited by 0

On the map: LLM agents and tool use → Web, search, and research agents

Abstract (arXiv)

Advances in large language models (LLMs) are rapidly transforming scientific work, yet empirical evidence on how these systems reshape research activities remains limited. We report a mixed-methods pilot evaluation of an AI-orchestrated research workflow in which a human researcher coordinated multiple LLM-based agents to perform data extraction, corpus construction, artifact generation, and artifact evaluation. Using the generation and assessment of multiple-choice questions (MCQs) as a testbed, we collected 1,071 SAT Math MCQs and employed LLM agents to extract questions from PDFs, retrieve and convert open textbooks into structured representations, align each MCQ with relevant textbook content, generate new MCQs under specified difficulty and cognitive levels, and evaluate both original and generated MCQs using a 24-criterion quality framework. Across all evaluations, average MCQ quality was high. However, criterion-level analysis and equivalence testing show that generated MCQs are not fully comparable to expert-vetted baseline questions. Strict similarity (24/24 criteria equivalent) was never achieved. Persistent gaps concentrated in skill depth, cognitive engagement, difficulty calibration, and metadata alignment, while surface-level qualities, such as grammar fluency, clarity options, no duplicates, were consistently strong. Beyond MCQ outcomes, the study documents a labor shift. The researcher's work moved from ``authoring items'' toward specification, orchestration, verification, and governance. Formalizing constraints, designing rubrics, building validation loops, recovering from tool failures, and auditing provenance constituted the primary activities. We discuss implications for the future of scientific work, including emerging ``AI research operations'' skills required for AI-empowered research pipelines.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Web, search, and research agents, one of 16 directions in LLM agents and tool use. Builds on 18 of its 43 references that are in the map. First posted on arXiv 21 Feb 2026.

Builds on

18 of 43 references.

References · 18

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  4. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
  5. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  6. 2022Large Language Models are Zero-Shot Reasonerscited by 4,374
  7. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,182
  8. 2021On the Opportunities and Risks of Foundation Modelscited by 4,033
  9. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,373
  10. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,276
  11. 2024A Comprehensive Survey of Scientific Large Language Models and Their Applications in Scientific Discoverycited by 96
  12. 2025From Automation to Autonomy: A Survey on Large Language Models in Scientific Discoverycited by 93
6 more
  1. 2025LLM Generated Persona is a Promise with a Catchcited by 81
  2. 2024Generative Students: Using LLM-Simulated Student Profiles to Support Question Item Evaluationcited by 58
  3. 2025LLMs Reproduce Human Purchase Intent via Semantic Similarity Elicitation of Likert Ratingscited by 12
  4. 2025Multiple-Choice Question Generation Using Large Language Models: Methodology and Educator Insightscited by 8
  5. 2026Take Out Your Calculators: Estimating the Real Difficulty of Question Items with LLM Student Simulationscited by 5
  6. 2026Cognitively Diverse Multiple-Choice Question Generation: A Hybrid Multi-Agent Framework with Large Language Modelscited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Generating AI Literacy MCQs: A Multi-Agent LLM Approach
  2. 2025EducationQ: Evaluating LLMs' Teaching Capabilities Through Multi-Agent Dialogue Framework
  3. 2026FirstResearch: Auditable Question Formation for LLM Scientific Discovery Agents
  4. 2023CoQuest: Exploring Research Question Co-Creation with an LLM-based Agent
  5. 2023ChatEval: Towards Better LLM-based Evaluators through Multi-Agent Debate
  6. 2024The Future of Learning in the Age of Generative AI: Automated Question Generation and Assessment with Large Language Models
  7. 2024TestAgent: Automatic Benchmarking and Exploratory Interaction for Evaluating LLMs in Vertical Domains
  8. 2024ResearchAgent: Iterative Research Idea Generation over Scientific Literature with Large Language Models