mlsanityData to 7 Oct 2026

Towards Teachable Reasoning Systems: Using a Dynamic Memory of User Feedback for Continual System Improvement

arXiv 2204.13074 ↗First posted 27 Apr 2022cs.CL cs.AIcited by 37

On the map: LLM reasoning → Test-time compute and efficient thinking

Abstract (arXiv)

Our goal is a teachable reasoning system for question-answering (QA), where a user can interact with faithful answer explanations, and correct its errors so that the system improves over time. Our approach is to augment a QA model with a dynamic memory of user feedback, containing user-supplied corrections to erroneous model beliefs that users identify during interaction. Retrievals from memory are used as additional context for QA, to help avoid previous mistakes in similar new situations - a novel application of memory-based continuous learning. With simulated feedback, we find that our system (called TeachMe) continually improves with time, and without model retraining, requiring feedback on only 25% of training examples to reach within 1% of the upper-bound (feedback on all examples). Similarly, in experiments with real users, we observe a similar trend, with performance improving by over 15% on a hidden test set after teaching. This suggests new opportunities for using frozen language models in an interactive setting where users can inspect, debug, and correct the model's beliefs, leading to improved system's performance over time.

Key facts. Cited by 37 papers in the map (data through 7 Oct 2026). Placed in Test-time compute and efficient thinking, one of 17 directions in LLM reasoning. Builds on 37 of its 56 references that are in the map. First posted on arXiv 27 Apr 2022.

Builds on

37 of 56 references.

References · 37

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  3. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  4. 2019The Curious Case of Neural Text Degenerationcited by 2,881
  5. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,114
  6. 2019Language Models as Knowledge Bases?cited by 1,922
  7. 2022Competition-Level Code Generation with AlphaCodecited by 1,427
  8. 2018Continual Lifelong Learning with Neural Networks: A Reviewcited by 1,313
  9. 2021Show Your Work: Scratchpads for Intermediate Computation with Language Modelscited by 871
  10. 2021Learning To Retrieve Prompts for In-Context Learningcited by 641
  11. 2021Editing Factual Knowledge in Language Modelscited by 548
  12. 2021Fast Model Editing at Scalecited by 491
25 more
  1. 2020UnifiedQA: Crossing Format Boundaries With a Single QA Systemcited by 475
  2. 2019COMET: Commonsense Transformers for Automatic Knowledge Graph Constructioncited by 450
  3. 2019Learning the Difference that Makes a Difference with Counterfactually-Augmented Datacited by 428
  4. 2020ProofWriter: Generating Implications, Proofs, and Abductive Statements over Natural Languagecited by 309
  5. 2017Right for the Right Reasons: Training Differentiable Models by Constraining their Explanationscited by 293
  6. 2020Transformers as Soft Reasoners over Languagecited by 265
  7. 2020Unsupervised Commonsense Question Answering with Self-Talkcited by 161
  8. 2021Explaining Answers with Entailment Treescited by 148
  9. 2021Teach Me to Explain: A Review of Datasets for Explainable Natural Language Processingcited by 126
  10. 2018Transforming Question Answering Datasets Into Natural Language Inference Datasetscited by 99
  11. 2019QuaRTz: An Open-Domain Dataset of Qualitative Relationship Questionscited by 89
  12. 2021Do Language Models Have Beliefs? Methods for Detecting, Updating, and Visualizing Model Beliefscited by 79
  13. 2021BeliefBank: Adding Memory to a Pre-Trained Language Model for a Systematic Notion of Beliefcited by 56
  14. 2022Rethinking Explainability as a Dialogue: A Practitioner's Perspectivecited by 52
  15. 2019Recommendation as a Communication Game: Self-Supervised Bot-Play for Goal-oriented Dialoguecited by 44
  16. 2021General-Purpose Question-Answering with Macawcited by 40
  17. 2021Learning to Repair: Repairing model output errors after deployment using a dynamic memory of feedbackcited by 35
  18. 2019From 'F' to 'A' on the N.Y. Regents Science Exams: An Overview of the Aristo Projectcited by 33
  19. 2021Flexible Generation of Natural Language Deductionscited by 28
  20. 2022A Rationale-Centric Framework for Human-in-the-loop Machine Learningcited by 27
  21. 2020Leap-Of-Thought: Teaching Pre-Trained Models to Systematically Reason Over Implicit Knowledgecited by 24
  22. 2020Paragraph-level Commonsense Transformers with Recurrent Memorycited by 23
  23. 2018Learning to Attend On Essential Terms: An Enhanced Retriever-Reader Model for Open-domain Question Answeringcited by 12
  24. 2020Braid: Weaving Symbolic and Neural Knowledge into Coherent Logical Explanationscited by 11
  25. 2021Self-Supervised Bot Play for Conversational Recommendation with Justificationscited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 5; 2023: 14; 2024: 9; 2025: 6; 2026: 3.

37 papers.

Sort:

By area:

  1. 2023Fine-Grained Human Feedback Gives Better Rewards for Language Model Trainingcited by 399
  2. 2023Automatically Correcting Large Language Models: Surveying the landscape of diverse self-correction strategiescited by 236
  3. 2024From Persona to Personalization: A Survey on Role-Playing Language Agentscited by 206
  4. 2023Let's Sample Step by Step: Adaptive-Consistency for Efficient Reasoning and Coding with LLMscited by 111
  5. 2023CLIN: A Continually Learning Language Agent for Rapid Task Adaptation and Generalizationcited by 88
  6. 2023Put Your Money Where Your Mouth Is: Evaluating Strategic Planning and Execution of LLM Agents in an Auction Arenacited by 78
  7. 2024In-context Learning with Retrieved Demonstrations for Language Models: A Surveycited by 69
  8. 2025A Survey of Personalized Large Language Models: Progress and Future Directionscited by 65
  9. 2024A Survey on Deep Learning for Theorem Provingcited by 55
  10. 2023LLM-based Medical Assistant Personalization with Short- and Long-Term Memory Coordinationcited by 46
  11. 2022Entailer: Answering Questions with Faithful and Truthful Chains of Reasoningcited by 38
  12. 2024TravelAgent: An AI Assistant for Personalized Travel Planningcited by 38
25 more
  1. 2023How Do Large Language Models Capture the Ever-changing World Knowledge? A Review of Recent Advancescited by 33
  2. 2024Personalized LLM Response Generation with Parameterized Memory Injectioncited by 33
  3. 2023Logical Reasoning over Natural Language as Knowledge Representation: A Surveycited by 32
  4. 2023Incremental Learning of Humanoid Robot Behavior from Natural Interaction and Large Language Modelscited by 19
  5. 2023Learning Deductive Reasoning from Synthetic Corpus based on Formal Logiccited by 18
  6. 2022NELLIE: A Neuro-Symbolic Inference Engine for Grounded, Compositional, and Explainable Reasoningcited by 16
  7. 2023Faithful Question Answering with Monte-Carlo Planningcited by 16
  8. 2024Position: Towards Bidirectional Human-AI Alignmentcited by 15
  9. 2022Summarization Programs: Interpretable Abstractive Summarization with Neural Modular Treescited by 14
  10. 2023InterroLang: Exploring NLP Models and Datasets through Dialogue-based Explanationscited by 11
  11. 2023RoleCraft-GLM: Advancing Personalized Role-Playing in Large Language Modelscited by 9
  12. 2022Impact of Feedback Type on Explanatory Interactive Learningcited by 7
  13. 2024AI, Meet Human: Learning Paradigms for Hybrid Decision Making Systemscited by 7
  14. 2024RAM: Towards an Ever-Improving Memory System by Learning from Communicationscited by 6
  15. 2023An Interactive Framework for Profiling News Media Sourcescited by 4
  16. 2024Measuring, Modeling, and Helping People Account for Privacy Risks in Online Self-Disclosures with AIcited by 4
  17. 2025Post-Incorporating Code Structural Knowledge into Pretrained Models via ICL for Code Translationcited by 3
  18. 2026CARD: Cluster-level Adaptation with Reward-guided Decoding for Personalized Text Generationcited by 2
  19. 2026Neuromem: A Granular Decomposition of the Streaming Lifecycle in External Memory for LLMscited by 2
  20. 2025ValuePilot: A Two-Phase Framework for Value-Driven Decision-Makingcited by 1
  21. 2022Towards Language-driven Scientific AIcited by 0
  22. 2025Failures Are the Stepping Stones to Success: Enhancing Few-Shot In-Context Learning by Leveraging Negative Samplescited by 0
  23. 2025Handling Open-Vocabulary Constructs in Formalizing Specifications: Retrieval-Augmented Parsing with Expert Knowledgecited by 0
  24. 2025ValuePilot: A Two-Phase Framework for Value-Driven Decision-Makingcited by 0
  25. 2026Do Implicit Personalization and Explicit Styles Conflict? PsPLUG: A Lightweight Plug-in for Balancing Personalization and Style in Customized LLMscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026What Am I Missing? Question-Answering as Hidden State Probing
  2. 2025Socratic Students: Teaching Language Models to Learn by Asking Questions
  3. 2026CoLearn: An Agentic Tutor that Learns its Learner in a Human--AI Co-Learning Loop
  4. 2025Evaluating Long-Term Memory for Long-Context Question Answering
  5. 2025Continual Learning of Domain Knowledge from Human Feedback in Text-to-SQL
  6. 2025T2: An Adaptive Test-Time Scaling Strategy for Contextual Question Answering
  7. 2026A Verifier-Guided Explainable Reasoning Framework with Gold-Anchored QLoRA, Task-Aware Mixture-of-Experts, and Group-Relative RLVR
  8. 2025First Try Matters: Revisiting the Role of Reflection in Reasoning Models