mlsanityData to 7 Oct 2026

Reward Learning from Multiple Feedback Types

arXiv 2502.21038 ↗First posted 28 Feb 2025cs.LG cs.AIcited by 9

On the map: LLM agents and tool use → Reinforcement learning for LLM agents

also in 1 more
Abstract (arXiv)

Learning rewards from preference feedback has become an important tool in the alignment of agentic models. Preference-based feedback, often implemented as a binary comparison between multiple completions, is an established method to acquire large-scale human feedback. However, human feedback in other contexts is often much more diverse. Such diverse feedback can better support the goals of a human annotator, and the simultaneous use of multiple sources might be mutually informative for the learning process or carry type-dependent biases for the reward learning process. Despite these potential benefits, learning from different feedback types has yet to be explored extensively. In this paper, we bridge this gap by enabling experimentation and evaluating multi-type feedback in a broad set of environments. We present a process to generate high-quality simulated feedback of six different types. Then, we implement reward models and downstream RL training for all six feedback types. Based on the simulated feedback, we investigate the use of types of feedback across ten RL environments and compare them to pure preference-based baselines. We show empirically that diverse types of feedback can be utilized and lead to strong reward modeling performance. This work is the first strong indicator of the potential of multi-type feedback for RLHF.

Key facts. Cited by 9 papers in the map (data through 7 Oct 2026). Placed in Reinforcement learning for LLM agents, one of 16 directions in LLM agents and tool use. Builds on 26 of its 50 references that are in the map. First posted on arXiv 28 Feb 2025.

Builds on

26 of 50 references.

References · 26

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2017Axiomatic Attribution for Deep Networkscited by 3,350
  4. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  5. 2019Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learningcited by 1,168
  6. 2023RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedbackcited by 615
  7. 2019Leveraging Procedural Generation to Benchmark Reinforcement Learningcited by 478
  8. 2023A Survey of Reinforcement Learning from Human Feedbackcited by 259
  9. 2019Extrapolating Beyond Suboptimal Demonstrations via Inverse Reinforcement Learning from Observationscited by 256
  10. 2017Deep TAMER: Interactive Agent Shaping in High-Dimensional State Spacescited by 122
  11. 2020Reward-rational (implicit) choice: A unifying formalism for reward learningcited by 119
  12. 2022Models of human preference for learning reward functionscited by 54
14 more
  1. 2021A Survey on Interactive Reinforcement Learning: Design Principles and Open Challengescited by 41
  2. 2024Gymnasium: A Standard Interface for Reinforcement Learning Environmentscited by 35
  3. 2022imitation: Clean Imitation Learning Implementationscited by 34
  4. 2022The Effect of Modeling Human Rationality Level on Learning Rewards from Multiple Feedback Typescited by 26
  5. 2023Reinforcement Learning from Diverse Human Preferencescited by 24
  6. 2022How to talk so AI will learn: Instructions, descriptions, and autonomycited by 22
  7. 2022Unified Learning from Demonstrations, Corrections, and Preferences during Physical Human-Robot Interactioncited by 18
  8. 2023RLHF-Blender: A Configurable Interactive Interface for Learning from Diverse Human Feedbackcited by 15
  9. 2018Reward Learning from Narrated Demonstrationscited by 13
  10. 2024Enhancing Robotic Manipulation with AI Feedback from Multimodal Large Language Modelscited by 10
  11. 2022Time-Efficient Reward Learning via Visually Assisted Cluster Rankingcited by 8
  12. 2021SocialAI: Benchmarking Socio-Cognitive Abilities in Deep Reinforcement Learning Agentscited by 7
  13. 2022Linguistic communication as (inverse) reward designcited by 6
  14. 2022How to Enable Uncertainty Estimation in Proximal Policy Optimizationcited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 5; 2026: 4.

9 papers.

  1. 2025The Lighthouse of Language: Enhancing LLM Agents via Critique-Guided Improvementcited by 19
  2. 2025Alignment and Safety in Large Language Models: Safety Mechanisms, Training Paradigms, and Emerging Challengescited by 14
  3. 2025DxHF: Providing High-Quality Human Feedback for LLM Alignment via Interactive Decompositioncited by 3
  4. 2025From Noise to Signal to Selbstzweck: Reframing Human Label Variation in the Era of Post-training in NLPcited by 3
  5. 2025Reward Models are Metrics in a Trench Coatcited by 2
  6. 2026Reward Learning through Ranking Mean Squared Errorcited by 0
  7. 2026Building Intelligent User Interfaces for Human-AI Alignmentcited by 0
  8. 2026MAVRL: Learning Reward Functions from Multiple Feedback Types with Amortized Variational Inferencecited by 0
  9. 2026Multi-Modal, Multi-Environment Machine Teaching for Robust Reward Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Mapping out the Space of Human Feedback for Reinforcement Learning: A Conceptual Framework
  2. 2026Reinforcement Learning from Human Feedback: A Statistical Perspective
  3. 2024Personalizing Reinforcement Learning from Human Feedback with Variational Preference Learning
  4. 2025Reinforcement Learning from Multi-level and Episodic Human Feedback
  5. 2023The History and Risks of Reinforcement Learning and Human Feedback
  6. 2025Capturing Individual Human Preferences with Reward Features
  7. 2024RLHF from Heterogeneous Feedback via Personalization and Preference Aggregation
  8. 2025RLBFF: Binary Flexible Feedback to bridge between Human Feedback & Verifiable Rewards