mlsanityData to 14 Oct 2026

From Problem-Solving to Teaching Problem-Solving: Aligning LLMs with Pedagogy using Reinforcement Learning

arXiv 2505.15607 ↗First posted 21 May 2025cs.CL cs.AIcited by 17

Not on the map. Why?

Abstract (arXiv)

Large language models (LLMs) can transform education, but their optimization for direct question-answering often undermines effective pedagogy which requires strategically withholding answers. To mitigate this, we propose an online reinforcement learning (RL)-based alignment framework that can quickly adapt LLMs into effective tutors using simulated student-tutor interactions by emphasizing pedagogical quality and guided problem-solving over simply giving away answers. We use our method to train a 7B parameter tutor model without human annotations which reaches similar performance to larger proprietary models like LearnLM. We introduce a controllable reward weighting to balance pedagogical support and student solving accuracy, allowing us to trace the Pareto frontier between these two objectives. Our models better preserve reasoning capabilities than single-turn SFT baselines and can optionally enhance interpretability through thinking tags that expose the model's instructional planning.

Key facts. Cited by 17 papers in the map (data through 14 Oct 2026). Builds on 32 of its 46 references that are in the map. First posted on arXiv 21 May 2025.

Builds on

32 of 46 references.

References · 32

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2017Proximal Policy Optimization Algorithmscited by 15,215
  3. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  4. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  5. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  6. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  7. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  8. 2023Let's Verify Step by Stepcited by 3,789
  9. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,298
  10. 2024Tulu 3: Pushing Frontiers in Open Language Model Post-Trainingcited by 1,053
  11. 2015Sequence Level Training with Recurrent Neural Networkscited by 576
  12. 20218-bit Optimizers via Block-wise Quantizationcited by 403
20 more
  1. 2025RAGEN: Understanding Self-Evolution in LLM Agents via Multi-Turn Reinforcement Learningcited by 386
  2. 2024Capabilities of Gemini Models in Medicinecited by 237
  3. 2023Reward Model Ensembles Help Mitigate Overoptimizationcited by 231
  4. 2023MathDial: A Dialogue Tutoring Dataset with Rich Pedagogical Properties Grounded in Math Reasoning Problemscited by 131
  5. 2025Big-Math: A Large-Scale, High-Quality Math Dataset for Reinforcement Learning in Language Modelscited by 109
  6. 2025Gold-medalist Performance in Solving Olympiad Geometry with AlphaGeometry2cited by 97
  7. 2022The AI Teacher Test: Measuring the Pedagogical Ability of Blender and GPT-3 in Educational Dialoguescited by 79
  8. 2024Building Math Agents with Multi-Turn Iterative Preference Learningcited by 66
  9. 2024Unifying AI Tutor Evaluation: An Evaluation Taxonomy for Pedagogical Ability Assessment of LLM-Powered AI Tutorscited by 61
  10. 2023Bridging the Novice-Expert Gap via Models of Decision-Making: A Case Study on Remediating Math Mistakescited by 52
  11. 2023CLASS: A Design Framework for building Intelligent Tutoring Systems based on Learning Science principlescited by 48
  12. 2022Automatic Generation of Socratic Subquestions for Teaching Math Word Problemscited by 43
  13. 2025Training LLM-based Tutors to Improve Student Learning Outcomes in Dialoguescited by 43
  14. 2024Stepwise Verification and Remediation of Student Reasoning Errors with Large Language Model Tutorscited by 41
  15. 2023Opportunities and Challenges in Neural Dialog Tutoringcited by 33
  16. 2024Regressing the Relative Future: Efficient Policy Optimization for Multi-turn RLHFcited by 30
  17. 2024Book2Dial: Generating Teacher-Student Interactions from Textbooks for Cost-Effective Development of Educational Chatbotscited by 25
  18. 2024Towards the Pedagogical Steering of Large Language Models for Tutoring: A Case Study with Modeling Productive Failurecited by 24
  19. 2025Beyond Final Answers: Evaluating Large Language Models for Math Tutoringcited by 23
  20. 2024Pedagogical Alignment of Large Language Modelscited by 19

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 15.

17 papers.

Sort:

By area:

  1. 2026Quantifying the Utility of User Simulators for Building Collaborative LLM Assistantscited by 10
  2. 2025ClinTutor-R1: Advancing Scalable and Robust One-to-Many Alignment in Clinical Socratic Educationcited by 6
  3. 2026Letting Tutor Personas Speak Up for LLMs: Learning Steering Vectors from Dialogue via Preference Optimizationcited by 4
  4. 2026Rewarding How Models Think Pedagogically: Integrating Pedagogical Reasoning and Thinking Rewards for LLMs in Educationcited by 3
  5. 2026DeepTutor: Towards Agentic Personalized Tutoringcited by 2
  6. 2025Cultivating Helpful, Personalized, and Creative AI Tutors: A Framework for Pedagogical Alignment using Reinforcement Learningcited by 1
  7. 2026Teaching Language Models How to Code Like Learners: Conversational Serialization for Student Simulationcited by 1
  8. 2026Hey Chat, Can You Teach Me? Structuring Socratic Dialogue for Human Learning in the Wildcited by 1
  9. 2026StudentSim: Training LLM-based Student Simulatorscited by 1
  10. 2026Sycophancy is an Educational Safety Risk: Why LLM Tutors Need Sycophancy Benchmarkscited by 0
  11. 2026Context-Masked Truncated Reasoning Audits for Answer-Key Dependence in LLM Tutorscited by 0
  12. 2026HiFloat4 Format for End-To-End Reinforcement Learning Post-Training of Large Language Modelscited by 0
5 more
  1. 2026Auditable Release Control for Pedagogical Leakage in LLM Tutorscited by 0
  2. 2026ToST: A Tree-of-Thought Socratic Teaching Framework for Multi-Path Guidance and Parallel Thinkingcited by 0
  3. 2026Socrates went Nuclear: Comparing Interaction Strategies for AI systems in a Learning Context using Brain Sensingcited by 0
  4. 2026DICE: Decoupling Capability from Intervention Necessity in LLM Tutoringcited by 0
  5. 2026RegulaTutor: A Theory-Informed Framework for Long-Term Personalized Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Application-Driven Pedagogical Knowledge Optimization of Open-Source LLMs via Reinforcement Learning and Supervised Fine-Tuning
  2. 2026PEARL: Training Socratic Tutors with Pedagogically Aligned Reinforcement Learning
  3. 2025Efficient RL for optimizing conversation level outcomes with an LLM-based tutor
  4. 2025Learning Like Humans: Advancing LLM Reasoning Capabilities via Adaptive Difficulty Curriculum Learning and Expert-Guided Self-Reformulation
  5. 2025Teaching LLMs for Step-Level Automatic Math Correction via Reinforcement Learning
  6. 2026The Tutoring Effectiveness Index: Predicting LLM Math Tutor Quality from Four Conversation Signals
  7. 2026Towards Pedagogically Aligned LLM Tutors for Math Mistake Remediation
  8. 2025Self-Evolving Curriculum for LLM Reasoning