mlsanityData to 7 Oct 2026

Efficiently Aligning Language Models with Online Natural Language Feedback

arXiv 2605.04356 ↗First posted 5 May 2026cs.LG cs.AIcited by 0

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Reinforcement learning with verifiable rewards has been used to elicit impressive performance from language models in many domains. But, broadly beneficial deployments of AI may require us to train models with strong capabilities in "fuzzy", hard-to-supervise domains. In this paper, we develop methods to align language models in fuzzy domains where human experts are still able to provide high-quality supervision signal, but only for a small number of model outputs, using online natural language feedback. Specifically, we train models by iteratively optimizing against proxy reward signals, stopping at the point of over-optimization, collecting fresh expert supervision, and updating the proxy reward. We construct proxy reward models from language models using in-context learning (ICL) and fine-tuning. We test our methods by eliciting creative writing and alignment research capabilities in Qwen3-8B and Haiku 4.5 respectively. For Qwen3-8B, ICL methods recover up to 35% of performance with 50x fewer expert samples, while fine-tuning methods recover 80% with up to 20x fewer samples and 100% with 3x fewer samples. For Haiku 4.5, ICL methods recover up to 35% of performance with 30x fewer samples, and fine-tuning methods recover 100% with 10x fewer samples. Our results suggest that online natural language feedback can substantially improve the data efficiency of expert supervision.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 41 of its 47 references that are in the map. First posted on arXiv 5 May 2026.

Builds on

41 of 47 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024Direct Language Model Alignment from Online AI Feedback7 Feb
  2. 2023Data-Efficient Alignment of Large Language Models with Human Feedback Through Natural Language24 Nov
  3. 2022Training language models to follow instructions with human feedback4 Mar

Other references · 38

  1. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  2. 2025Qwen3 Technical Reportcited by 8,195
  3. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  4. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,146
  5. 2017Deep reinforcement learning from human preferencescited by 3,846
  6. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  7. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  8. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scalecited by 2,678
  9. 2020Learning to summarize from human feedbackcited by 2,562
  10. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  11. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,056
  12. 2023Towards Understanding Sycophancy in Language Modelscited by 988
26 more
  1. 2023Large Language Models as Optimizerscited by 830
  2. 2023Prometheus: Inducing Fine-grained Evaluation Capability in Language Modelscited by 558
  3. 2025GEPA: Reflective Prompt Evolution Can Outperform Reinforcement Learningcited by 449
  4. 2023Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervisioncited by 445
  5. 2018Scalable agent alignment via reward modeling: a research directioncited by 438
  6. 2023Fine-Grained Human Feedback Gives Better Rewards for Language Model Trainingcited by 399
  7. 2025Rubrics as Rewards: Reinforcement Learning Beyond Verifiable Domainscited by 328
  8. 2025Reinforcement Learning with Verifiable Rewards Implicitly Incentivizes Correct Reasoning in Base LLMscited by 276
  9. 2024Optimizing Instructions and Demonstrations for Multi-Stage Language Model Programscited by 270
  10. 2024Debating with More Persuasive LLMs Leads to More Truthful Answerscited by 255
  11. 2024TextGrad: Automatic "Differentiation" via Textcited by 250
  12. 2023Reward Model Ensembles Help Mitigate Overoptimizationcited by 225
  13. 2022Measuring Progress on Scalable Oversight for Large Language Modelscited by 212
  14. 2024DeepSeek-Prover-V1.5: Harnessing Proof Assistant Feedback for Reinforcement Learning and Monte-Carlo Tree Searchcited by 189
  15. 2024Generative Reward Modelscited by 148
  16. 2024Scaling Laws for Reward Model Overoptimization in Direct Alignment Algorithmscited by 139
  17. 2024LLM Critics Help Catch LLM Bugscited by 138
  18. 2024Language Models Learn to Mislead Humans via RLHFcited by 118
  19. 2023Confronting Reward Model Overoptimization with Constrained RLHFcited by 96
  20. 2024On scalable oversight with weak LLMs judging strong LLMscited by 94
  21. 2024LLM-Rubric: A Multidimensional, Calibrated Approach to Automated Evaluation of Natural Language Textscited by 81
  22. 2025ResearchRubrics: A Benchmark of Prompts and Rubrics For Evaluating Deep Research Agentscited by 80
  23. 2022Training Language Models with Language Feedbackcited by 48
  24. 2025Chasing the Tail: Effective Rubric-based Reward Modeling for Large Language Model Post-Trainingcited by 46
  25. 2025Feedback Descent: Open-Ended Text Optimization via Pairwise Comparisoncited by 42
  26. 2024Reinforcement Learning from Human Feedback with Active Queriescited by 39

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Rethinking the Role of Proxy Rewards in Language Model Alignment
  2. 2024Aligning Large Language Models via Fine-grained Supervision
  3. 2023Aligning Large Language Models through Synthetic Feedback
  4. 2026Theoretical Limits of Language Model Alignment
  5. 2023Language Model Alignment with Elastic Reset
  6. 2024Self-Exploring Language Models: Active Preference Elicitation for Online Alignment
  7. 2025Reusing Embeddings: Reproducible Reward Model Research in Large Language Model Alignment without GPUs
  8. 2023SALMON: Self-Alignment with Instructable Reward Models