mlsanityData to 14 Oct 2026

Improving Multimodal Interactive Agents with Reinforcement Learning from Human Feedback

arXiv 2211.11602 ↗First posted 21 Nov 2022cs.LG cs.HC cs.MAcited by 26

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

An important goal in artificial intelligence is to create agents that can both interact naturally with humans and learn from their feedback. Here we demonstrate how to use reinforcement learning from human feedback (RLHF) to improve upon simulated, embodied agents trained to a base level of competency with imitation learning. First, we collected data of humans interacting with agents in a simulated 3D world. We then asked annotators to record moments where they believed that agents either progressed toward or regressed from their human-instructed goal. Using this annotation data we leveraged a novel method - which we call "Inter-temporal Bradley-Terry" (IBT) modelling - to build a reward model that captures human judgments. Agents trained to optimise rewards delivered from IBT reward models improved with respect to all of our metrics, including subsequent human judgment during live interactions with agents. Altogether our results demonstrate how one can successfully leverage human judgments to improve agent behaviour, allowing us to use reinforcement learning in complex, embodied domains without programmatic reward functions. Videos of agent behaviour may be found at https://youtu.be/v_Z9F2_eKk4.

Key facts. Cited by 26 papers in the map (data through 14 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 24 of its 49 references that are in the map. First posted on arXiv 21 Nov 2022.

Builds on

24 of 49 references.

Closest ancestors · 4

Same direction, newest first

  1. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback12 Apr
  2. 2019Fine-Tuning Language Models from Human Preferences18 Sep
  3. 2018Reward learning from human preferences and demonstrations in Atari15 Nov
  4. 2017Deep reinforcement learning from human preferences12 Jun

Other references · 20

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2022Training language models to follow instructions with human feedbackcited by 15,971
  5. 2020Scaling Laws for Neural Language Modelscited by 6,586
  6. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  7. 2021Zero-Shot Text-to-Image Generationcited by 4,288
  8. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,864
  9. 2020Learning to summarize from human feedbackcited by 2,588
  10. 2022LaMDA: Language Models for Dialog Applicationscited by 1,220
  11. 2018IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architecturescited by 883
  12. 2022A Generalist Agentcited by 808
8 more
  1. 2022Improving alignment of dialogue agents via targeted human judgementscited by 531
  2. 2022Multi-Game Decision Transformerscited by 214
  3. 2017Interactive Learning from Policy-Dependent Human Feedbackcited by 158
  4. 2019Synthetic Data for Deep Learningcited by 113
  5. 2020Imitating Interactive Intelligencecited by 55
  6. 2021Creating Multimodal Interactive Agents with Imitation and Self-Supervised Learningcited by 40
  7. 2022Scaling Laws for a Multi-Agent Reinforcement Learning Modelcited by 38
  8. 2021Recent Advances in Leveraging Human Guidance for Sequential Decision-Making Taskscited by 13

Built on by

Cited by year

Citing papers per year: 2022: 1; 2023: 10; 2024: 9; 2025: 6.

26 papers.

Sort:

By area:

  1. 2023Tool Learning with Foundation Modelscited by 302
  2. 2023A Survey of Reinforcement Learning from Human Feedbackcited by 264
  3. 2023Reward Model Ensembles Help Mitigate Overoptimizationcited by 231
  4. 2023Vision-Language Models as Success Detectorscited by 142
  5. 2023Is RLHF More Difficult than Standard RL?cited by 96
  6. 2024RLHF from Heterogeneous Feedback via Personalization and Preference Aggregationcited by 74
  7. 2024Scaling Instructable Agents Across Many Simulated Worldscited by 71
  8. 2024Gen-Drive: Enhancing Diffusion Generative Driving Policies with Reward Modeling and Reinforcement Learning Fine-tuningcited by 42
  9. 2025Position: Multimodal Large Language Models Can Significantly Advance Scientific Reasoningcited by 33
  10. 2025Reward Models in Deep Reinforcement Learning: A Surveycited by 25
  11. 2023Reinforcement Learning with Human Feedback for Realistic Traffic Simulationcited by 21
  12. 2024PokerGPT: An End-to-End Lightweight Solver for Multi-Player Texas Hold'em via Large Language Modelcited by 20
14 more
  1. 2023Mitigating Open-Vocabulary Caption Hallucinationscited by 19
  2. 2023Perspectives on the Social Impacts of Reinforcement Learning with Human Feedbackcited by 15
  3. 2023Learning a Universal Human Prior for Dexterous Manipulation from Human Preferencecited by 10
  4. 2023Solving NLP Problems through Human-System Collaboration: A Discussion-based Approachcited by 5
  5. 2025A Shared Low-Rank Adaptation Approach to Personalized RLHFcited by 4
  6. 2025Interactive Groupwise Comparison for Reinforcement Learning from Human Feedbackcited by 4
  7. 2024On The Global Convergence Of Online RLHF With Neural Parametrizationcited by 3
  8. 2024Behavioural Cloning in VizDoomcited by 2
  9. 2024Evolutionary Retrofittingcited by 2
  10. 2022Inclusive Artificial Intelligencecited by 1
  11. 2024Aligning Agents like Large Language Modelscited by 1
  12. 2025Conditional Multi-Stage Failure Recovery for Embodied Agentscited by 1
  13. 2024ICC: Quantifying Image Caption Concreteness for Multimodal Dataset Curationcited by 0
  14. 2025Inducing Robustness in a 2 Dimensional Direct Preference Optimization Paradigmcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Mapping out the Space of Human Feedback for Reinforcement Learning: A Conceptual Framework
  2. 2025RFTF: Reinforcement Fine-tuning for Embodied Agents with Temporal Feedback
  3. 2017Actor-Critic Reinforcement Learning with Simultaneous Human Control and Feedback
  4. 2021PEBBLE: Feedback-Efficient Interactive Reinforcement Learning via Relabeling Experience and Unsupervised Pre-training
  5. 2025An Introduction to Deep Reinforcement and Imitation Learning
  6. 2025SHARPIE: A Modular Framework for Reinforcement Learning and Human-AI Interaction Experiments
  7. 2025Reward Learning from Multiple Feedback Types
  8. 2025Reinforcement Learning from Multi-level and Episodic Human Feedback