METIS: Mentoring Engine for Thoughtful Inquiry & Solutions
On the map: LLM reasoning → Reasoning benchmarks and evaluation
Abstract (arXiv)
Many students lack access to expert research mentorship. We ask whether an AI mentor can move undergraduates from an idea to a paper. We build METIS, a tool-augmented, stage-aware assistant with literature search, curated guidelines, methodology checks, and memory. We evaluate METIS against GPT-5 and Claude Sonnet 4.5 across six writing stages using LLM-as-a-judge pairwise preferences, student-persona rubrics, short multi-turn tutoring, and evidence/compliance checks. On 90 single-turn prompts, LLM judges preferred METIS to Claude Sonnet 4.5 in 71% and to GPT-5 in 54%. Student scores (clarity/actionability/constraint-fit; 90 prompts x 3 judges) are higher across stages. In multi-turn sessions (five scenarios/agent), METIS yields slightly higher final quality than GPT-5. Gains concentrate in document-grounded stages (D-F), consistent with stage-aware routing and groundings failure modes include premature tool routing, shallow grounding, and occasional stage misclassification.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 11 of its 22 references that are in the map. First posted on arXiv 19 Jan 2026.
Builds on
11 of 22 references.
Closest ancestors · 2
Same direction, newest first
- 2022ReAct: Synergizing Reasoning and Acting in Language Models6 Oct
- 2022MRKL Systems: A modular, neuro-symbolic architecture that combines large language models, external knowledge sources and discrete reasoning1 May
Other references · 9
- 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
- 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,072
- 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,064
- 2023AgentBench: Evaluating LLMs as Agentscited by 1,211
- 2023GAIA: a benchmark for General AI Assistantscited by 1,084
- 2024Judging the Judges: A Systematic Study of Position Bias in LLM-as-a-Judgecited by 249
- 2024Self-Preference Bias in LLM-as-a-Judgecited by 248
- 2024Can You Trust LLM Judgments? Reliability of LLM-as-a-Judgecited by 52
- 2024Explaining Length Bias in LLM-Based Preference Evaluationscited by 48
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026Prober.ai: Gated Inquiry-Based Feedback via LLM-Constrained Personas for Argumentative Writing Development
- 2026PaperMentor: A Human-Centered Multi-Agent Writing Tutor for AI Research Papers on Overleaf
- 2025Chrysalis: A Unified System for Comparing Active Teaching and Passive Learning with AI Agents in Education
- 2026EduClaw-Bench: A Long-Horizon Benchmark for Pedagogical LLM Agents with Simulated Learners
- 2025Automated Formative Feedback for Short-form Writing: An LLM-Driven Approach and Adoption Analysis
- 2023Bridging Learnersourcing and AI: Exploring the Dynamics of Student-AI Collaborative Feedback Generation
- 2026PIVOT: Preference-based Intervention Vectors for Pedagogical Tutor Steering
- 2025Rethinking Search: A Study of University Students' Perspectives on Using LLMs and Traditional Search Engines in Academic Problem Solving