mlsanityData to 7 Oct 2026

Rethinking LLM-Judged Helpfulness as a Pedagogy Signal: A Pre-Registered Audit Across Tutor Models

arXiv 2607.28128 ↗First posted 30 Jul 2026cs.CL cs.AI cs.CYcited by 1

On the map: LLM reasoning → Reasoning benchmarks and evaluation

Abstract (arXiv)

LLM tutoring poses a measurement problem: can a general-purpose helpfulness rubric distinguish direct answer-giving from pedagogical guidance? We audit this signal in a pre-registered study. Within each of three tutor bases, we compare conversational and pedagogical policies instantiated with the same underlying model and paired with one fixed weak simulated student. Deterministic detectors measure answer leakage and next-turn independent work. Claude Opus 4.8 is the frozen, condition-blind primary judge. After the Opus scores were fixed, GPT-5.6 Sol was prospectively specified for a post hoc robustness audit of the same 1,179 confirmatory answer-phase tutor turns under the frozen helpfulness and pedagogy rubrics. On the primary base under Opus, the policies do not differ significantly in helpfulness but are perfectly rank-separated under the pedagogy rubric (Cliff's |δ|=0.10 vs. 1.0). Across the two judges, pedagogy contrasts retain their direction where detected, whereas the helpfulness ordering is judge-contingent, reversing between judges on two of three bases. In an Opus-only ablation, seven primary-base policies span 2.3 points in mean judged pedagogy within a 0.25-point band of mean judged helpfulness. Separately, answer-revealing turns are followed by less independent student work on every base, a result that is judge-invariant by construction. In this controlled setting, general-purpose helpfulness is not a reliable pedagogy signal. Tutor evaluation should pair pedagogy-targeted rubrics with deterministic process measures.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 7 of its 26 references that are in the map. First posted on arXiv 30 Jul 2026.

Builds on

7 of 26 references.

References · 7

  1. 2024The Llama 3 Herd of Modelscited by 14,192
  2. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  3. 2024Self-Preference Bias in LLM-as-a-Judgecited by 248
  4. 2024Towards Responsible Development of Generative AI for Education: An Evaluation-Driven Approachcited by 53
  5. 2024LearnLM: Improving Gemini for Learningcited by 24
  6. 2026Simulated Students in Tutoring Dialogues: Substance or Illusion?cited by 14
  7. 2026Towards Valid Student Simulation with Large Language Modelscited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.

Similar, not cited

Close in wording, no citation either way

  1. 2026Confirming Correct, Missing the Rest: LLM Tutoring Agents Struggle Where Feedback Matters Most
  2. 2025Large Language Models Approach Expert Pedagogical Quality in Math Tutoring but Differ in Instructional and Linguistic Profiles
  3. 2026Rewarding How Models Think Pedagogically: Integrating Pedagogical Reasoning and Thinking Rewards for LLMs in Education
  4. 2026The Tutoring Effectiveness Index: Predicting LLM Math Tutor Quality from Four Conversation Signals
  5. 2026Teaching a Large Language Model Tutor to Withhold the Answer: A Supervisor Architecture and an Evidence-Driven Method for Tuning Socratic Behavior ↗
  6. 2026PIVOT: Preference-based Intervention Vectors for Pedagogical Tutor Steering
  7. 2025MathTutorBench: A Benchmark for Measuring Open-ended Pedagogical Capabilities of LLM Tutors
  8. 2026Micro-level AI Feedback Features and Student Responses in Consecutive LLM Tutoring Interactions ↗