mlsanityData to 7 Oct 2026

Retrospective Learning from Interactions

arXiv 2410.13852 ↗First posted 17 Oct 2024cs.CL cs.AI cs.CVcited by 6

Not on the map. Why?

Abstract (arXiv)

Multi-turn interactions between large language models (LLMs) and users naturally include implicit feedback signals. If an LLM responds in an unexpected way to an instruction, the user is likely to signal it by rephrasing the request, expressing frustration, or pivoting to an alternative task. Such signals are task-independent and occupy a relatively constrained subspace of language, allowing the LLM to identify them even if it fails on the actual task. We introduce ReSpect, a method to learn from such signals in past interactions via retrospection without additional annotations. We deploy ReSpect in a new multimodal interaction scenario, where humans instruct a multimodal LLM to solve an abstract reasoning task with a combinatorial solution space. Through thousands of interactions with humans, we show how ReSpect gradually improves task completion rate from 31% to 82%, all without any external annotation.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Builds on 35 of its 53 references that are in the map. First posted on arXiv 17 Oct 2024.

Builds on

35 of 53 references.

References · 35

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  4. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  5. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,317
  6. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  7. 2019When Does Label Smoothing Help?cited by 985
  8. 2022Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learningcited by 886
  9. 2024Self-Rewarding Language Modelscited by 654
  10. 2017Ray: A Distributed Framework for Emerging AI Applicationscited by 516
  11. 2023Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervisioncited by 445
  12. 2024Training Language Models to Self-Correct via Reinforcement Learningcited by 405
23 more
  1. 2024What matters when building vision-language models?cited by 321
  2. 2024Quiet-STaR: Language Models Can Teach Themselves to Think Before Speakingcited by 292
  3. 2023Training Language Models with Language Feedback at Scalecited by 125
  4. 2019Learning from Dialogue after Deployment: Feed Yourself, Chatbot!cited by 102
  5. 2017Mapping Instructions and Visual Observations to Actions with Reinforcement Learningcited by 80
  6. 2024Prover-Verifier Games improve legibility of LLM outputscited by 79
  7. 2024Aligning LLM Agents by Learning Latent Preference from User Editscited by 66
  8. 2024Selective Reflection-Tuning: Student-Selected Data Recycling for LLM Instruction-Tuningcited by 57
  9. 2016Dialogue Learning With Human-In-The-Loopcited by 46
  10. 2022Abstract Visual Reasoning with Tangram Shapescited by 43
  11. 2023Benchmarking and Improving Generator-Validator Consistency of Language Modelscited by 43
  12. 2020Learning Rewards from Linguistic Feedbackcited by 41
  13. 2019Characterizing the dynamics of learning in repeated reference gamescited by 34
  14. 2021Learning to Give Checkable Answers with Prover-Verifier Gamescited by 29
  15. 2019Continual adaptation for efficient machine communicationcited by 28
  16. 2022Continual Learning for Instruction Following from Realtime Feedbackcited by 21
  17. 2021Continual Learning for Grounded Instruction Generation by Observing Human Following Behaviorcited by 19
  18. 2020Empirica: a virtual lab for high-throughput macro-level experimentscited by 16
  19. 2023Leveraging Implicit Feedback from Deployment Data in Dialoguecited by 14
  20. 2023Continually Improving Extractive QA via Human Feedbackcited by 11
  21. 2021Analysis of Language Change in Collaborative Instruction Followingcited by 9
  22. 2023Learning From Free-Text Human Feedback -- Collect New Datasets Or Extend Existing Ones?cited by 7
  23. 2024CoGen: Learning from Feedback with Coupled Comprehension and Generationcited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 2; 2026: 4.

6 papers.

  1. 2025The Era of Real-World Human Interaction: RL from User Conversationscited by 13
  2. 2025Playpen: An Environment for Exploring Learning Through Conversational Interactioncited by 2
  3. 2026CharacterFlywheel: Scaling Iterative Improvement of Engaging and Steerable LLMs in Productioncited by 1
  4. 2026Learning from 53.6K Real-World Developer Edits of AI-Generated Codecited by 1
  5. 2026Principled Fine-tuning of LLMs from User-Edits: A Medley of Preference, Supervision, and Rewardcited by 0
  6. 2026Different Feedback, Different Updates: Selective Self-Learning from User Interactions for Large Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025InterMT: Multi-Turn Interleaved Preference Alignment with Human Feedback
  2. 2025CUPID: Evaluating Personalized and Contextualized Alignment of LLMs from Interactions
  3. 2026Words & Weights: Streamlining Multi-Turn Interactions via Co-Adaptation
  4. 2026ProactiveBench: Benchmarking Proactiveness in Multimodal Large Language Models
  5. 2024Grounding Multimodal Large Language Models in Actions
  6. 2026Beyond the Assistant Turn: User Turn Generation as a Probe of Interaction Awareness in Language Models
  7. 2024Talk Less, Interact Better: Evaluating In-context Conversational Adaptation in Multimodal LLMs
  8. 2026PRACTICE: From Experience to Expertise in Self-Evolving Embodied Agents