mlsanityData to 14 Oct 2026

Let Me Teach You: Pedagogical Foundations of Feedback for Language Models

arXiv 2307.00279 ↗First posted 1 Jul 2023cs.CLcited by 9

On the map: LLM alignment / RLHF / preference optimisation → Human values and AI feedback alignment

Abstract (arXiv)

Natural Language Feedback (NLF) is an increasingly popular mechanism for aligning Large Language Models (LLMs) to human preferences. Despite the diversity of the information it can convey, NLF methods are often hand-designed and arbitrary, with little systematic grounding. At the same time, research in learning sciences has long established several effective feedback models. In this opinion piece, we compile ideas from pedagogy to introduce FELT, a feedback framework for LLMs that outlines various characteristics of the feedback space, and a feedback content taxonomy based on these variables, providing a general mapping of the feedback space. In addition to streamlining NLF designs, FELT also brings out new, unexplored directions for research in NLF. We make our taxonomy available to the community, providing guides and examples for mapping our categorizations to future research.

Key facts. Cited by 9 papers in the map (data through 14 Oct 2026). Placed in Human values and AI feedback alignment, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 48 of its 114 references that are in the map. First posted on arXiv 1 Jul 2023.

Builds on

48 of 114 references.

Closest ancestors · 8

Same direction, newest first

  1. 2023Bridging the Gap: A Survey on Integrating (Human) Feedback for Natural Language Generation1 May
  2. 2023Pretraining Language Models with Human Preferences16 Feb
  3. 2023The Capacity for Moral Self-Correction in Large Language Models15 Feb
  4. 2022Constitutional AI: Harmlessness from AI Feedback15 Dec
  5. 2022When Life Gives You Lemons, Make Cherryade: Converting Feedback from Bad Responses into Good Labels28 Oct
  6. 2022Learning New Skills after Deployment: Improving open-domain internet-driven dialogue with human feedback5 Aug
  7. 2022Training Language Models with Language Feedback29 Apr
  8. 2020Learning to summarize from human feedback2 Sep

Other references · 40

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2022Training language models to follow instructions with human feedbackcited by 15,971
  4. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  5. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,276
  6. 2017Deep reinforcement learning from human preferencescited by 3,912
  7. 2023Let's Verify Step by Stepcited by 3,789
  8. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  9. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,442
  10. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,724
  11. 2019Fine-Tuning Language Models from Human Preferencescited by 2,150
  12. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,095
28 more
  1. 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,055
  2. 2023Teaching Large Language Models to Self-Debugcited by 988
  3. 2023OpenAssistant Conversations -- Democratizing Large Language Model Alignmentcited by 721
  4. 2019ELI5: Long Form Question Answeringcited by 575
  5. 2023How Close is ChatGPT to Human Experts? Comparison Corpus, Evaluation, and Detectioncited by 481
  6. 2023Fine-Grained Human Feedback Gives Better Rewards for Language Model Trainingcited by 401
  7. 2022Self-critiquing models for assisting human evaluatorscited by 380
  8. 2021Entity-Based Knowledge Conflicts in Question Answeringcited by 356
  9. 2022The Effects of Reward Misspecification: Mapping and Mitigating Misaligned Modelscited by 328
  10. 2022Teaching language models to support answers with verified quotescited by 292
  11. 2021Understanding Dataset Difficulty with V-Usable Informationcited by 291
  12. 2021Recursively Summarizing Books with Human Feedbackcited by 290
  13. 2022Generating Sequences by Learning to Self-Correctcited by 277
  14. 2022Is Reinforcement Learning (Not) for Natural Language Processing: Benchmarks, Baselines, and Building Blocks for Natural Language Policy Optimizationcited by 246
  15. 2023REFINER: Reasoning Feedback on Intermediate Representationscited by 205
  16. 2022ROSCOE: A Suite of Metrics for Scoring Step-by-Step Reasoningcited by 177
  17. 2023Chain of Hindsight Aligns Language Models with Feedbackcited by 148
  18. 2019Learning from Dialogue after Deployment: Feed Yourself, Chatbot!cited by 102
  19. 2022PEER: A Collaborative Language Modelcited by 86
  20. 2023Context-faithful Prompting for Large Language Modelscited by 80
  21. 2018Can Neural Machine Translation be Improved with User Feedback?cited by 72
  22. 2016Dialogue Learning With Human-In-The-Loopcited by 46
  23. 2021Learning to Repair: Repairing model output errors after deployment using a dynamic memory of feedbackcited by 35
  24. 2022Fixing Model Bugs with Natural Language Patchescited by 33
  25. 2023Inference-Time Policy Adapters (IPA): Tailoring Extreme-Scale LMs without Fine-tuningcited by 33
  26. 2016Dialog-based Language Learningcited by 21
  27. 2022Simulating Bandit Learning from User Feedback for Extractive Question Answeringcited by 14
  28. 2023MISMATCH: Fine-grained Evaluation of Machine-generated Text with Mismatch Error Typescited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 2; 2025: 5; 2026: 2.

9 papers.

  1. 2024Unifying AI Tutor Evaluation: An Evaluation Taxonomy for Pedagogical Ability Assessment of LLM-Powered AI Tutorscited by 61
  2. 2024Bringing Generative AI to Adaptive Learning in Educationcited by 24
  3. 2025User Feedback in Human-LLM Dialogues: A Lens to Understand Users But Noisy as a Learning Signalcited by 19
  4. 2025Position: LLMs Can be Good Tutors in English Educationcited by 6
  5. 2025A Review of Incorporating Psychological Theories in LLMscited by 6
  6. 2025Critique-Guided Distillation for Robust Reasoning via Refinementcited by 2
  7. 2026Investigating Learner-Aware Design of LLM-Generated Educational Feedbackcited by 1
  8. 2025FEANEL: A Benchmark for Fine-Grained Error Analysis in K-12 English Writingcited by 0
  9. 2026How Well Can LLMs Simulate Real Learner Evaluations of Educational Feedback?cited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023The Past, Present and Better Future of Feedback Learning in Large Language Models for Subjective Human Preferences and Values
  2. 2026Evaluating Feedback Focus and Pedagogical Adaptivity in LLM-Generated Feedback on Student Writing
  3. 2023System-Level Natural Language Feedback
  4. 2023Human Feedback is not Gold Standard
  5. 2026SEFORA: Student Essays with Feedback Corpus and LLM Feedback Evaluation Framework
  6. 2022Curriculum: A Broad-Coverage Benchmark for Linguistic Phenomena in Natural Language Understanding
  7. 2024The PRISM Alignment Dataset: What Participatory, Representative and Individualised Human Feedback Reveals About the Subjective and Multicultural Alignment of Large Language Models
  8. 2025Feedback Indicators: The Alignment between Llama and a Teacher in Language Learning