mlsanityData to 7 Oct 2026

Improving Socratic Question Generation using Data Augmentation and Preference Optimization

arXiv 2403.00199 ↗First posted 1 Mar 2024cs.CL cs.CY cs.LGcited by 15

On the map: LLM alignment / RLHF / preference optimisation → Process reward models for reasoning

Abstract (arXiv)

The Socratic method is a way of guiding students toward solving a problem independently without directly revealing the solution to the problem. Although this method has been shown to significantly improve student learning outcomes, it remains a complex labor-intensive task for instructors. Large language models (LLMs) can be used to augment human effort by automatically generating Socratic questions for students. However, existing methods that involve prompting these LLMs sometimes produce invalid outputs, e.g., those that directly reveal the solution to the problem or provide irrelevant or premature questions. To alleviate this problem, inspired by reinforcement learning with AI feedback (RLAIF), we first propose a data augmentation method to enrich existing Socratic questioning datasets with questions that are invalid in specific ways. Next, we propose a method to optimize open-source LLMs such as LLama 2 to prefer ground-truth questions over generated invalid ones, using direct preference optimization (DPO). Our experiments on a Socratic questions dataset for student code debugging show that a DPO-optimized 7B LLama 2 model can effectively avoid generating invalid questions, and as a result, outperforms existing state-of-the-art prompting methods.

Key facts. Cited by 15 papers in the map (data through 7 Oct 2026). Placed in Process reward models for reasoning, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 27 of its 43 references that are in the map. First posted on arXiv 1 Mar 2024.

Builds on

27 of 43 references.

References · 27

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  5. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  6. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  7. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  8. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  9. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  10. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,579
  11. 2020Learning to summarize from human feedbackcited by 2,562
  12. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,205
15 more
  1. 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,157
  2. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  3. 2023Code Llama: Open Foundation Models for Codecited by 2,105
  4. 2023RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedbackcited by 615
  5. 2022Is Reinforcement Learning (Not) for Natural Language Processing: Benchmarks, Baselines, and Building Blocks for Natural Language Policy Optimizationcited by 244
  6. 2022Automatic Generation of Programming Exercises and Code Explanations using Large Language Modelscited by 124
  7. 2024CodeAid: Evaluating a Classroom Deployment of an LLM-based Programming Assistant that Balances Student and Educator Needscited by 81
  8. 2023EduChat: A Large-Scale Language Model-based Chatbot System for Intelligent Educationcited by 80
  9. 2018Reliability and Learnability of Human Bandit Feedback for Sequence-to-Sequence Reinforcement Learningcited by 57
  10. 2022Automatic Generation of Socratic Subquestions for Teaching Math Word Problemscited by 42
  11. 2021Math Word Problem Generation with Mathematical Consistency and Problem Context Constraintscited by 27
  12. 2023Can Language Models Employ the Socratic Method? Experiments with Code Debuggingcited by 14
  13. 2023Improving Reading Comprehension Question Generation with Data Augmentation and Overgenerate-and-rankcited by 12
  14. 2024Using Large Language Models for Student-Code Guided Test Case Generation in Computer Science Educationcited by 4
  15. 2024OpineBot: Class Feedback Reimagined Using a Conversational LLMcited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 5; 2025: 7; 2026: 3.

15 papers.

Sort:

By area:

  1. 2025Training LLM-based Tutors to Improve Student Learning Outcomes in Dialoguescited by 40
  2. 2024Improving Automated Distractor Generation for Math Multiple-choice Questions with Overgenerate-and-rankcited by 12
  3. 2024Test Case-Informed Knowledge Tracing for Open-ended Coding Taskscited by 7
  4. 2025LookAlike: Consistent Distractor Generation in Math MCQscited by 7
  5. 2025Opportunities and Challenges of LLMs in Education: An NLP Perspectivecited by 6
  6. 2025Discerning minds or generic tutors? Evaluating instructional guidance capabilities in Socratic LLMscited by 5
  7. 2024Generating Feedback-Ladders for Logical Errors in Programming using Large Language Modelscited by 3
  8. 2024Persona-SQ: A Personalized Suggested Question Generation Framework For Real-world Documentscited by 3
  9. 2024Interpreting Latent Student Knowledge Representations in Programming Assignmentscited by 2
  10. 2025Inverse Knowledge Search over Verifiable Reasoning: Synthesizing a Scientific Encyclopedia from a Long Chains-of-Thought Knowledge Basecited by 2
  11. 2025FEAT: A Preference Feedback Dataset through a Cost-Effective Auto-Generation and Labeling Framework for English AI Tutoringcited by 1
  12. 2026Teaching Language Models How to Code Like Learners: Conversational Serialization for Student Simulationcited by 1
3 more
  1. 2026Your Students Don't Use LLMs Like You Wish They Didcited by 1
  2. 2025ACE-RLHF: Automated Code Evaluation and Socratic Feedback Generation Tool using Large Language Models and Reinforcement Learning with Human Feedbackcited by 0
  3. 2026Incentives Of EdTech: A Systematic Review Of EduNLP Researchcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Socratic Students: Teaching Language Models to Learn by Asking Questions
  2. 2026Beyond Direct Answering: Aligning Educational LLMs as Socratic Guides via Heuristic Reinforcement Learning
  3. 2025Self-Questioning Language Models
  4. 2024The Future of Learning in the Age of Generative AI: Automated Question Generation and Assessment with Large Language Models
  5. 2024Boosting Large Language Models with Socratic Method for Conversational Mathematics Teaching
  6. 2024Unleashing LLM Reasoning Capability via Scalable Question Synthesis from Scratch
  7. 2025Socratic-Zero : Bootstrapping Reasoning via Data-Free Agent Co-evolution
  8. 2024Learning to Ask Informative Questions: Enhancing LLMs with Preference Optimization and Expected Information Gain