mlsanityData to 14 Oct 2026

Formalizing Learning from Language Feedback with Provable Guarantees

arXiv 2506.10341 ↗First posted 12 Jun 2025cs.LG cs.CLcited by 13

Not on the map. Why?

Abstract (arXiv)

Interactively learning from observation and language feedback is an increasingly studied area driven by the emergence of large language model (LLM) agents. Despite impressive empirical demonstrations, so far a principled framing of these decision problems remains lacking. We formalize the Learning from Language Feedback (LLF) problem, assert sufficient assumptions to enable learning despite latent rewards, and introduce textittransfer eluder dimension as a measure to characterize the hardness of LLF. We formalize the intuition that information in the language feedback governs the learning complexity, and demonstrate cases where learning from rich language feedback can be exponentially faster than learning from reward. We develop a no-regret algorithm, called textttHELiX, that provably solves LLF problems through sequential interactions, with performance guarantees that scale with the transfer eluder dimension. Across several empirical domains, we show that textttHELiX performs well even when repeatedly prompting LLMs does not work reliably. Our contributions mark an important step towards designing principled interactive learning algorithms using generic language feedback.

Key facts. Cited by 13 papers in the map (data through 14 Oct 2026). Builds on 58 of its 84 references that are in the map. First posted on arXiv 12 Jun 2025.

Builds on

58 of 84 references.

References · 58

  1. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  2. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  3. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  4. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,330
  5. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,276
  6. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,592
  7. 2023Voyager: An Open-Ended Embodied Agent with Large Language Modelscited by 2,191
  8. 2023Improving Factuality and Reasoning in Language Models through Multiagent Debatecited by 1,784
  9. 2024A Survey on LLM-as-a-Judgecited by 1,423
  10. 2023Teaching Large Language Models to Self-Debugcited by 988
  11. 2021An Explanation of In-context Learning as Implicit Bayesian Inferencecited by 938
  12. 2022What learning algorithm is in-context learning? Investigations with linear modelscited by 684
38 more
  1. 2017A Tutorial on Thompson Samplingcited by 433
  2. 2025Cognitive Behaviors that Enable Self-Improving Reasoners, or, Four Habits of Highly Effective STaRscited by 377
  3. 2024Can LLMs Generate Novel Research Ideas? A Large-Scale Human Study with 100+ NLP Researcherscited by 363
  4. 2022Large Language Models are Better Reasoners with Self-Verificationcited by 307
  5. 2025The Illusion of Thinking: Understanding the Strengths and Limitations of Reasoning Models via the Lens of Problem Complexitycited by 302
  6. 2014A Survey of Multi-Objective Sequential Decision-Makingcited by 242
  7. 2023ReMax: A Simple, Effective, and Efficient Reinforcement Learning Method for Aligning Large Language Modelscited by 187
  8. 2023Chain of Hindsight Aligns Language Models with Feedbackcited by 148
  9. 2023Supervised Pretraining Can Learn In-Context Reinforcement Learningcited by 139
  10. 2022DiffCSE: Difference-based Contrastive Learning for Sentence Embeddingscited by 132
  11. 2022Language Models as Agent Modelscited by 132
  12. 2023Training Language Models with Language Feedback at Scalecited by 128
  13. 2025Optimizing Test-Time Compute via Meta Reinforcement Fine-Tuningcited by 123
  14. 2014Reinforcement and Imitation Learning via Interactive No-Regret Learningcited by 118
  15. 2024WorldCoder, a Model-Based LLM Agent: Building World Models by Writing Code and Interacting with the Environmentcited by 107
  16. 2023RL4F: Generating Natural Language Feedback with Reinforcement Learning for Repairing Model Outputscited by 105
  17. 2023LongEval: Guidelines for Human Evaluation of Faithfulness in Long-form Summarizationcited by 96
  18. 2013The Optimal Reward Baseline for Gradient-Based Reinforcement Learningcited by 85
  19. 2016Multi-Objective Deep Reinforcement Learningcited by 83
  20. 2020Logic-Guided Data Augmentation and Regularization for Consistent Question Answeringcited by 78
  21. 2024Can large language models explore in-context?cited by 76
  22. 2017Learning with Latent Languagecited by 75
  23. 2023How Do Transformers Learn In-Context Beyond Simple Functions? A Case Study on Learning with Representationscited by 75
  24. 2024AutoGuide: Automated Generation and Selection of Context-Aware Guidelines for Large Language Model Agentscited by 74
  25. 2024Hypothesis Generation with Large Language Modelscited by 71
  26. 2024Trace is the Next AutoDiff: Generative Optimization with Rich Feedback, Execution Traces, and LLMscited by 69
  27. 2025Robin: A multi-agent system for automating scientific discoverycited by 62
  28. 2021Reinforcement Learning, Bit by Bitcited by 59
  29. 2022Improving Intrinsic Exploration with Language Abstractionscited by 49
  30. 2022Training Language Models with Language Feedbackcited by 49
  31. 2019An Information-Theoretic Approach to Minimax Regret in Partial Monitoringcited by 42
  32. 2025LLMs are Greedy Agents: Effects of RL Fine-tuning on Decision-Making Abilitiescited by 41
  33. 2014Learning to Win by Reading Manuals in a Monte-Carlo Frameworkcited by 39
  34. 2023Generalization to New Sequential Decision Making Tasks with In-Context Learningcited by 39
  35. 2024EVOLvE: Evaluating and Optimizing LLMs For In-Context Explorationcited by 34
  36. 2023LLF-Bench: Benchmark for Interactive Learning from Language Feedbackcited by 33
  37. 2022Large Language Models can Implement Policy Iterationcited by 31
  38. 2025Training a Generally Curious Agentcited by 31

50 most cited of 58 All 58 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 12.

13 papers.

Sort:

By area:

  1. 2026Expanding the Capabilities of Reinforcement Learning via Text Feedbackcited by 53
  2. 2026RetroAgent: From Solving to Evolving via Retrospective Dual Intrinsic Feedbackcited by 17
  3. 2026Understanding the Challenges in Iterative Generative Optimization with LLMscited by 8
  4. 2026Learning from Language Feedback via Variational Policy Distillationcited by 7
  5. 2025On the Limits of Test-Time Compute: Sequential Reward Filtering for Better Inferencecited by 6
  6. 2026DIVE: Unlocking Self-Improvement in Frozen Language Models Through Diversity-Driven Skill Evolutioncited by 5
  7. 2026Inference-time Unlearning Using Conformal Predictioncited by 2
  8. 2026Learning to Learn from Language Feedback with Social Meta-Learningcited by 1
  9. 2026Interaction-Grounded Learning for Contextual Markov Decision Processes with Personalized Feedbackcited by 0
  10. 2026Inductive Feedback for Mixed-Policy Distillationcited by 0
  11. 2026Learning What to Investigate Next: Meta-Reasoning for Long-Horizon Research Agentscited by 0
  12. 2026Constraint Tree Exploration for Learning from Language Feedbackcited by 0
1 more
  1. 2026Reflective Prompt Optimization of a Deep-Research Agent: A Topic-Disjoint, Multi-Seed Protocol for Measuring Transferable Gains and Prompt-Level Overfittingcited by 0