mlsanityData to 14 Oct 2026

CLARIFY: Contrastive Preference Reinforcement Learning for Untangling Ambiguous Queries

arXiv 2506.00388 ↗First posted 31 May 2025cs.LGcited by 3

On the map: Deep reinforcement learning methods → RL training of language and agent models

Abstract (arXiv)

Preference-based reinforcement learning (PbRL) bypasses explicit reward engineering by inferring reward functions from human preference comparisons, enabling better alignment with human intentions. However, humans often struggle to label a clear preference between similar segments, reducing label efficiency and limiting PbRL's real-world applicability. To address this, we propose an offline PbRL method: Contrastive LeArning for ResolvIng Ambiguous Feedback (CLARIFY), which learns a trajectory embedding space that incorporates preference information, ensuring clearly distinguished segments are spaced apart, thus facilitating the selection of more unambiguous queries. Extensive experiments demonstrate that CLARIFY outperforms baselines in both non-ideal teachers and real human feedback settings. Our approach not only selects more distinguished queries but also learns meaningful trajectory embeddings.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Placed in RL training of language and agent models, one of 15 directions in Deep reinforcement learning methods. Builds on 32 of its 39 references that are in the map. First posted on arXiv 31 May 2025.

Builds on

32 of 39 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024S-EPOA: Overcoming the Indistinguishability of Segments with Skill-Driven Preference-Based Reinforcement Learning22 Aug
  2. 2024RIME: Robust Preference-based Reinforcement Learning with Noisy Preferences27 Feb

Other references · 30

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  3. 2017Deep reinforcement learning from human preferencescited by 3,912
  4. 2020Exploring Simple Siamese Representation Learningcited by 2,866
  5. 2015Trust Region Policy Optimizationcited by 2,580
  6. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,427
  7. 2021Offline Reinforcement Learning with Implicit Q-Learningcited by 1,293
  8. 2019Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learningcited by 1,204
  9. 2018DeepMind Control Suitecited by 956
  10. 2017A Distributional Perspective on Reinforcement Learningcited by 765
  11. 2020CURL: Contrastive Unsupervised Representations for Reinforcement Learningcited by 760
  12. 2018QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulationcited by 743
18 more
  1. 2018Reward learning from human preferences and demonstrations in Ataricited by 316
  2. 2021PEBBLE: Feedback-Efficient Interactive Reinforcement Learning via Relabeling Experience and Unsupervised Pre-trainingcited by 302
  3. 2021B-Pref: Benchmarking Preference-Based Reinforcement Learningcited by 106
  4. 2021Generalized Decision Transformer for Offline Hindsight Information Matchingcited by 106
  5. 2022SURF: Semi-supervised Reward Learning with Data Augmentation for Feedback-efficient Preference-based Reinforcement Learningcited by 97
  6. 2023Preference Transformer: Modeling Human Preferences using Transformers for RLcited by 79
  7. 2020Active Preference-Based Gaussian Process Regression for Reward Learningcited by 71
  8. 2023Inverse Preference Learning: Preference-based RL without a Reward Functioncited by 68
  9. 2022Reward Uncertainty for Exploration in Preference-based Reinforcement Learningcited by 61
  10. 2023Benchmarks and Algorithms for Offline Preference-Based Reward Learningcited by 56
  11. 2024Learning Temporal Distances: Contrastive Successor Features Can Provide a Metric Structure for Decision-Makingcited by 56
  12. 2022Robust Task Representations for Offline Meta-Reinforcement Learning via Contrastive Learningcited by 35
  13. 2023Beyond Reward: Offline Preference-guided Policy Optimizationcited by 35
  14. 2023Reinforcement Learning from Diverse Human Preferencescited by 24
  15. 2023Survival Instinct in Offline Reinforcement Learningcited by 20
  16. 2025Episodic Novelty Through Temporal Distancecited by 16
  17. 2021Policy Information Capacity: Information-Theoretic Measure for Task Complexity in Deep Reinforcement Learningcited by 15
  18. 2024Listwise Reward Estimation for Offline Preference-based Reinforcement Learningcited by 14

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 1.

3 papers.

  1. 2025Preference-based Multi-Objective Reinforcement Learningcited by 7
  2. 2025STAIR: Addressing Stage Misalignment through Temporal-Aligned Preference Reinforcement Learningcited by 3
  3. 2026Multi-Agent Reinforcement Learning via Agent-Specific Preferencecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Query-Policy Misalignment in Preference-Based Reinforcement Learning
  2. 2026From Reward-Free Representations to Preferences: Rethinking Offline Preference-Based Reinforcement Learning
  3. 2025PB2: Preference Space Exploration via Population-Based Methods in Preference-Based Reinforcement Learning
  4. 2023Contrastive Preference Learning: Learning from Human Feedback without RL
  5. 2025SENIOR: Efficient Query Selection and Preference-Guided Exploration in Preference-based Reinforcement Learning
  6. 2024Multi-Type Preference Learning: Empowering Preference-Based Reinforcement Learning with Equal Preferences
  7. 2025Similarity as Reward Alignment: Robust and Versatile Preference-based Reinforcement Learning
  8. 2024Preference-Guided Reinforcement Learning for Efficient Exploration