mlsanityData to 14 Oct 2026

Time-Efficient Reward Learning via Visually Assisted Cluster Ranking

arXiv 2212.00169 ↗First posted 30 Nov 2022cs.LG cs.HCcited by 8

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

One of the most successful paradigms for reward learning uses human feedback in the form of comparisons. Although these methods hold promise, human comparison labeling is expensive and time consuming, constituting a major bottleneck to their broader applicability. Our insight is that we can greatly improve how effectively human time is used in these approaches by batching comparisons together, rather than having the human label each comparison individually. To do so, we leverage data dimensionality-reduction and visualization techniques to provide the human with a interactive GUI displaying the state space, in which the user can label subportions of the state space. Across some simple Mujoco tasks, we show that this high-level approach holds promise and is able to greatly increase the performance of the resulting agents, provided the same amount of human labeling time.

Key facts. Cited by 8 papers in the map (data through 14 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 29 of its 47 references that are in the map. First posted on arXiv 30 Nov 2022.

Builds on

29 of 47 references.

Closest ancestors · 4

Same direction, newest first

  1. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback12 Apr
  2. 2019Learning Human Objectives by Evaluating Hypothetical Behavior5 Dec
  3. 2018Reward learning from human preferences and demonstrations in Atari15 Nov
  4. 2017Deep reinforcement learning from human preferences12 Jun

Other references · 25

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  3. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  4. 2020Learning to summarize from human feedbackcited by 2,588
  5. 2020CURL: Contrastive Unsupervised Representations for Reinforcement Learningcited by 760
  6. 2017Learning Robust Rewards with Adversarial Inverse Reinforcement Learningcited by 490
  7. 2016Guided Cost Learning: Deep Inverse Optimal Control via Policy Optimizationcited by 406
  8. 2020Decoupling Representation Learning from Reinforcement Learningcited by 269
  9. 2019Extrapolating Beyond Suboptimal Demonstrations via Inverse Reinforcement Learning from Observationscited by 260
  10. 2017Reverse Curriculum Generation for Reinforcement Learningcited by 198
  11. 2017Deep TAMER: Interactive Agent Shaping in High-Dimensional State Spacescited by 123
  12. 2019End-to-End Robotic Reinforcement Learning without Reward Engineeringcited by 122
13 more
  1. 2018Playing hard exploration games by watching YouTubecited by 99
  2. 2018Learning Montezuma's Revenge from a Single Demonstrationcited by 80
  3. 2016Graying the black box: Understanding DQNscited by 68
  4. 2019Asking Easy Questions: A User-Friendly Approach to Active Reward Learningcited by 68
  5. 2019Learning Reward Functions by Integrating Human Demonstrations and Preferencescited by 64
  6. 2019Batch Active Learning Using Determinantal Point Processescited by 43
  7. 2016Watch This: Scalable Cost-Function Learning for Path Planning in Urban Environmentscited by 33
  8. 2019Risk-Aware Active Inverse Reinforcement Learningcited by 26
  9. 2018Towards Better Interpretability in Deep Q-Networkscited by 22
  10. 2021Increasing the Speed and Accuracy of Data LabelingThrough an AI Assisted Interfacecited by 21
  11. 2022Unified Learning from Demonstrations, Corrections, and Preferences during Physical Human-Robot Interactioncited by 19
  12. 2019Learning Representations by Humans, for Humanscited by 18
  13. 2021Learning Perceptual Concepts by Bootstrapping from Human Queriescited by 12

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 3; 2024: 1; 2025: 3; 2026: 1.

8 papers.

  1. 2023A Survey of Reinforcement Learning from Human Feedbackcited by 264
  2. 2025FDPP: Fine-tune Diffusion Policy with Human Preferencecited by 15
  3. 2023What Matters to You? Towards Visual Representation Alignment for Robot Learningcited by 14
  4. 2025Reward Learning from Multiple Feedback Typescited by 9
  5. 2024Mapping out the Space of Human Feedback for Reinforcement Learning: A Conceptual Frameworkcited by 5
  6. 2025Interactive Groupwise Comparison for Reinforcement Learning from Human Feedbackcited by 4
  7. 2023Contextual Bandits and Imitation Learning via Preference-Based Active Queriescited by 2
  8. 2026Building Intelligent User Interfaces for Human-AI Alignmentcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Scaling Reward Modeling without Human Supervision
  2. 2025Probing Preference Representations: A Multi-Dimensional Evaluation and Analysis Method for Reward Models
  3. 2025GRAM-R2: Self-Training Generative Foundation Reward Models for Reward Reasoning
  4. 2025Similarity as Reward Alignment: Robust and Versatile Preference-based Reinforcement Learning
  5. 2025Preference Learning with Response Time: Robust Losses and Guarantees
  6. 2022Reward Learning with Trees: Methods and Evaluation
  7. 2024CARMO: Dynamic Criteria Generation for Context-Aware Reward Modelling
  8. 2025GRAM: A Generative Foundation Reward Model for Reward Generalization