mlsanityData to 14 Oct 2026

Language Models Can Learn from Verbal Feedback Without Scalar Rewards

arXiv 2509.22638 ↗First posted 26 Sep 2025cs.CL cs.AI cs.LGcited by 18

Not on the map. Why?

Abstract (arXiv)

LLMs are often trained with RL from human or AI feedback, yet such methods typically compress nuanced feedback into scalar rewards, discarding much of their richness and inducing scale imbalance. We propose treating verbal feedback as a conditioning signal. Inspired by language priors in text-to-image generation, which enable novel outputs from unseen prompts, we introduce the feedback-conditional policy (FCP). FCP learns directly from response-feedback pairs, approximating the feedback-conditional posterior through maximum likelihood training on offline data. We further develop an online bootstrapping stage where the policy generates under positive conditions and receives fresh feedback to refine itself. This reframes feedback-driven learning as conditional generation rather than reward optimization, offering a more expressive way for LLMs to directly learn from verbal feedback. Our code is available at https://github.com/sail-sg/feedback-conditional-policy.

Key facts. Cited by 18 papers in the map (data through 14 Oct 2026). Builds on 63 of its 74 references that are in the map. First posted on arXiv 26 Sep 2025.

Builds on

63 of 74 references.

References · 63

  1. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  2. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  3. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  4. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,330
  5. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  6. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  7. 2024Qwen2.5 Technical Reportcited by 4,942
  8. 2025Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilitiescited by 3,796
  9. 2023Let's Verify Step by Stepcited by 3,789
  10. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  11. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,879
  12. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scalecited by 2,810
38 more
  1. 2019Fine-Tuning Language Models from Human Preferencescited by 2,150
  2. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 2,057
  3. 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,848
  4. 2018World Modelscited by 1,513
  5. 2024LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Modelscited by 1,464
  6. 2025Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learningcited by 1,444
  7. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,392
  8. 2025Understanding R1-Zero-Like Training: A Critical Perspectivecited by 1,345
  9. 2023Instruction-Following Evaluation for Large Language Modelscited by 1,228
  10. 2025Kimi k1.5: Scaling Reinforcement Learning with LLMscited by 1,210
  11. 2022Solving math word problems with process- and outcome-based feedbackcited by 901
  12. 2025AlphaEvolve: A coding agent for scientific and algorithmic discoverycited by 867
  13. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 782
  14. 2023RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignmentcited by 686
  15. 2023Prometheus: Inducing Fine-grained Evaluation Capability in Language Modelscited by 575
  16. 2025Process Reinforcement through Implicit Rewardscited by 504
  17. 2024Generative Verifiers: Reward Modeling as Next-Token Predictioncited by 480
  18. 2025Open-Reasoner-Zero: An Open Source Approach to Scaling Up Reinforcement Learning on the Base Modelcited by 461
  19. 2025RAGEN: Understanding Self-Evolution in LLM Agents via Multi-Turn Reinforcement Learningcited by 386
  20. 2025OpenThoughts: Data Recipes for Reasoning Modelscited by 304
  21. 2025VAPO: Efficient and Reliable Reinforcement Learning for Advanced Reasoning Taskscited by 285
  22. 2025SWE-smith: Scaling Data for Software Engineering Agentscited by 285
  23. 2025SWE-RL: Advancing LLM Reasoning via Reinforcement Learning on Open Software Evolutioncited by 263
  24. 2025Logic-RL: Unleashing LLM Reasoning with Rule-Based Reinforcement Learningcited by 262
  25. 2025Inference-Time Scaling for Generalist Reward Modelingcited by 239
  26. 2023TheoremQA: A Theorem-driven Question Answering datasetcited by 192
  27. 2025Light-R1: Curriculum SFT, DPO and RL for Long COT from Scratch and Beyondcited by 175
  28. 2024Generative Reward Modelscited by 148
  29. 2025SimpleTIR: End-to-End Reinforcement Learning for Multi-Turn Tool-Integrated Reasoningcited by 146
  30. 2024MAmmoTH2: Scaling Instructions from the Webcited by 145
  31. 2025General-Reasoner: Advancing LLM Reasoning Across All Domainscited by 135
  32. 2025Big-Math: A Large-Scale, High-Quality Math Dataset for Reinforcement Learning in Language Modelscited by 109
  33. 2024Critique-out-Loud Reward Modelscited by 106
  34. 2021Pretraining Representations for Data-Efficient Reinforcement Learningcited by 105
  35. 2025VerlTool: Towards Holistic Agentic Reinforcement Learning with Tool Usecited by 96
  36. 2025A Sober Look at Progress in Language Model Reasoning: Pitfalls and Paths to Reproducibilitycited by 93
  37. 2025Critique-GRPO: Advancing LLM Reasoning with Natural Language and Numerical Feedbackcited by 90
  38. 2024On Designing Effective RL Reward at Training Time for LLM Reasoningcited by 87

50 most cited of 63 All 63 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 16.

18 papers.

Sort:

By area:

  1. 2026Reinforcement Learning via Self-Distillationcited by 332
  2. 2026Privileged Information Distillation for Language Modelscited by 90
  3. 2026Expanding the Capabilities of Reinforcement Learning via Text Feedbackcited by 53
  4. 2026Aligning Language Models from User Interactionscited by 28
  5. 2025Moloch's Bargain: Emergent Misalignment When LLMs Compete for Audiencescited by 8
  6. 2026Learning from Language Feedback via Variational Policy Distillationcited by 7
  7. 2025DOCR-Inspector: Fine-Grained and Automated Evaluation of Document Parsing with VLMcited by 6
  8. 2026Illusions of Confidence? Diagnosing LLM Truthfulness via Neighborhood Consistencycited by 2
  9. 2026Hint-Guided Diversified Policy Optimization for LLM Reasoningcited by 2
  10. 2026MDForge: Agentic Molecular Dynamics Pipeline Design under Sparse Simulator Feedbackcited by 2
  11. 2026ALIVE: Awakening LLM Reasoning via Adversarial Learning and Instructive Verbal Evaluationcited by 1
  12. 2026Listening to the Echo: User-Reaction Aware Policy Optimization via Scalar-Verbal Hybrid Reinforcement Learningcited by 1
6 more
  1. 2026Introspective X Training: Feedback Conditioning Improves Scaling Across all LLM Training Stagescited by 1
  2. 2026A History-Aware Visually Grounded Critic for Computer Use Agentscited by 1
  3. 2026Co-Evolving Actor-Conditioned Critics for Non-Verifiable Generationcited by 1
  4. 2026Different Feedback, Different Updates: Selective Self-Learning from User Interactions for Large Language Modelscited by 0
  5. 2026The Rise of Verbal Reinforcement Learningcited by 0
  6. 2026Function-Space Temporal Self-Distillation for Policy Optimizationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Training Language Models with Language Feedback at Scale
  2. 2023Fine-Grained Human Feedback Gives Better Rewards for Language Model Training
  3. 2020Learning Rewards from Linguistic Feedback
  4. 2025Text2Grad: Reinforcement Learning from Natural Language Feedback
  5. 2023Interpreting Learned Feedback Patterns in Large Language Models
  6. 2026T5: Twin-Critic Training for Token-Level Thoughts in Reinforcement Mid-Training
  7. 2023Pretraining Language Models with Human Preferences
  8. 2025Reinforcement Learning from User Feedback