# Retrospective Learning from Interactions

arXiv: https://arxiv.org/abs/2410.13852 · HTML: https://mlsanity.com/paper/2410.13852/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 35 of its 53 references that are in the map.
- First posted on arXiv 17 Oct 2024.

## Builds on (most cited first, 30 of 35)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [Self-Refine: Iterative Refinement with Self-Feedback](https://mlsanity.com/paper/2303.17651/) (2023)
- [Constitutional AI: Harmlessness from AI Feedback](https://mlsanity.com/paper/2212.08073/) (2022)
- [When Does Label Smoothing Help?](https://mlsanity.com/paper/1906.02629/) (2019)
- [Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learning](https://mlsanity.com/paper/2205.05638/) (2022)
- [Self-Rewarding Language Models](https://mlsanity.com/paper/2401.10020/) (2024)
- [Ray: A Distributed Framework for Emerging AI Applications](https://mlsanity.com/paper/1712.05889/) (2017)
- [Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervision](https://mlsanity.com/paper/2312.09390/) (2023)
- [Training Language Models to Self-Correct via Reinforcement Learning](https://mlsanity.com/paper/2409.12917/) (2024)
- [What matters when building vision-language models?](https://mlsanity.com/paper/2405.02246/) (2024)
- [Quiet-STaR: Language Models Can Teach Themselves to Think Before Speaking](https://mlsanity.com/paper/2403.09629/) (2024)
- [Training Language Models with Language Feedback at Scale](https://mlsanity.com/paper/2303.16755/) (2023)
- [Learning from Dialogue after Deployment: Feed Yourself, Chatbot!](https://mlsanity.com/paper/1901.05415/) (2019)
- [Prover-Verifier Games improve legibility of LLM outputs](https://mlsanity.com/paper/2407.13692/) (2024)
- [Mapping Instructions and Visual Observations to Actions with Reinforcement Learning](https://mlsanity.com/paper/1704.08795/) (2017)
- [Aligning LLM Agents by Learning Latent Preference from User Edits](https://mlsanity.com/paper/2404.15269/) (2024)
- [Selective Reflection-Tuning: Student-Selected Data Recycling for LLM Instruction-Tuning](https://mlsanity.com/paper/2402.10110/) (2024)
- [Dialogue Learning With Human-In-The-Loop](https://mlsanity.com/paper/1611.09823/) (2016)
- [Benchmarking and Improving Generator-Validator Consistency of Language Models](https://mlsanity.com/paper/2310.01846/) (2023)
- [Abstract Visual Reasoning with Tangram Shapes](https://mlsanity.com/paper/2211.16492/) (2022)
- [Learning Rewards from Linguistic Feedback](https://mlsanity.com/paper/2009.14715/) (2020)
- [Characterizing the dynamics of learning in repeated reference games](https://mlsanity.com/paper/1912.07199/) (2019)
- [Learning to Give Checkable Answers with Prover-Verifier Games](https://mlsanity.com/paper/2108.12099/) (2021)
- [Continual adaptation for efficient machine communication](https://mlsanity.com/paper/1911.09896/) (2019)
- [Continual Learning for Instruction Following from Realtime Feedback](https://mlsanity.com/paper/2212.09710/) (2022)
- [Continual Learning for Grounded Instruction Generation by Observing Human Following Behavior](https://mlsanity.com/paper/2108.04812/) (2021)
- [Empirica: a virtual lab for high-throughput macro-level experiments](https://mlsanity.com/paper/2006.11398/) (2020)

## Built on by (most cited first, 6 of 6)

- [The Era of Real-World Human Interaction: RL from User Conversations](https://mlsanity.com/paper/2509.25137/) (2025)
- [Playpen: An Environment for Exploring Learning Through Conversational Interaction](https://mlsanity.com/paper/2504.08590/) (2025)
- [CharacterFlywheel: Scaling Iterative Improvement of Engaging and Steerable LLMs in Production](https://mlsanity.com/paper/2603.01973/) (2026)
- [Learning from 53.6K Real-World Developer Edits of AI-Generated Code](https://mlsanity.com/paper/2607.25130/) (2026)
- [Principled Fine-tuning of LLMs from User-Edits: A Medley of Preference, Supervision, and Reward](https://mlsanity.com/paper/2601.19055/) (2026)
- [Different Feedback, Different Updates: Selective Self-Learning from User Interactions for Large Language Models](https://mlsanity.com/paper/2608.09109/) (2026)

## Abstract (arXiv)

Multi-turn interactions between large language models (LLMs) and users naturally include implicit feedback signals. If an LLM responds in an unexpected way to an instruction, the user is likely to signal it by rephrasing the request, expressing frustration, or pivoting to an alternative task. Such signals are task-independent and occupy a relatively constrained subspace of language, allowing the LLM to identify them even if it fails on the actual task. We introduce ReSpect, a method to learn from such signals in past interactions via retrospection without additional annotations. We deploy ReSpect in a new multimodal interaction scenario, where humans instruct a multimodal LLM to solve an abstract reasoning task with a combinatorial solution space. Through thousands of interactions with humans, we show how ReSpect gradually improves task completion rate from 31% to 82%, all without any external annotation.