mlsanityData to 14 Oct 2026

TREND: Tri-teaching for Robust Preference-based Reinforcement Learning with Demonstrations

arXiv 2505.06079 ↗First posted 9 May 2025cs.RO cs.CVcited by 7

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

also in 1 more
Abstract (arXiv)

Preference feedback collected by human or VLM annotators is often noisy, presenting a significant challenge for preference-based reinforcement learning that relies on accurate preference labels. To address this challenge, we propose TREND, a novel framework that integrates few-shot expert demonstrations with a tri-teaching strategy for effective noise mitigation. Our method trains three reward models simultaneously, where each model views its small-loss preference pairs as useful knowledge and teaches such useful pairs to its peer network for updating the parameters. Remarkably, our approach requires as few as one to three expert demonstrations to achieve high performance. We evaluate TREND on various robotic manipulation tasks, achieving up to 90% success rates even with noise levels as high as 40%, highlighting its effective robustness in handling noisy preference feedback. Project page: https://shuaiyihuang.github.io/publications/TREND.

Key facts. Cited by 7 papers in the map (data through 14 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 41 of its 54 references that are in the map. First posted on arXiv 9 May 2025.

Builds on

41 of 54 references.

Closest ancestors · 9

Same direction, newest first

  1. 2023Vision-Language Models are Zero-Shot Reward Models for Reinforcement Learning19 Oct
  2. 2023Text2Reward: Reward Shaping with Language Models for Reinforcement Learning20 Sep
  3. 2023Reinforcement Learning from Diverse Human Preferences27 Jan
  4. 2022Reward Uncertainty for Exploration in Preference-based Reinforcement Learning24 May
  5. 2022SURF: Semi-supervised Reward Learning with Data Augmentation for Feedback-efficient Preference-based Reinforcement Learning18 Mar
  6. 2021PEBBLE: Feedback-Efficient Interactive Reinforcement Learning via Relabeling Experience and Unsupervised Pre-training9 Jun
  7. 2020Active Preference-Based Gaussian Process Regression for Reward Learning6 May
  8. 2018Reward learning from human preferences and demonstrations in Atari15 Nov
  9. 2017Deep reinforcement learning from human preferences12 Jun

Other references · 32

  1. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  2. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,509
  3. 2019Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learningcited by 1,204
  4. 2018Co-teaching: Robust Training of Deep Neural Networks with Extremely Noisy Labelscited by 1,059
  5. 2020Learning from Noisy Labels with Deep Neural Networks: A Surveycited by 582
  6. 2024TraceVLA: Visual Trace Prompting Enhances Spatial-Temporal Awareness for Generalist Robotic Policiescited by 304
  7. 2023Reward Design with Language Modelscited by 261
  8. 2020Does label smoothing mitigate label noise?cited by 204
  9. 2023Guiding Pretraining in Reinforcement Learning with Large Language Modelscited by 187
  10. 2024RL-VLM-F: Reinforcement Learning from Vision Language Foundation Model Feedbackcited by 156
  11. 2021B-Pref: Benchmarking Preference-Based Reinforcement Learningcited by 106
  12. 2023Motif: Intrinsic Motivation from Artificial Intelligence Feedbackcited by 102
20 more
  1. 2018Batch Active Preference-Based Learning of Reward Functionscited by 87
  2. 2020Denoising Implicit Feedback for Recommendationcited by 80
  3. 2023Preference Transformer: Modeling Human Preferences using Transformers for RLcited by 79
  4. 2023TACO: Temporal Latent Action-Driven Contrastive Loss for Visual Reinforcement Learningcited by 57
  5. 2022LMPriors: Pre-Trained Language Models as Task-Specific Priorscited by 56
  6. 2019Dynamic Context Correspondence Network for Semantic Alignmentcited by 43
  7. 2024RIME: Robust Preference-based Reinforcement Learning with Noisy Preferencescited by 39
  8. 2023Towards Scalable Neural Representation for Diverse Videoscited by 37
  9. 2024AutoHallusion: Automatic Generation of Hallucination Benchmarks for Vision-Language Modelscited by 32
  10. 2022Learning Semantic Correspondence with Sparse Annotationscited by 31
  11. 2024PRISE: LLM-Style Sequence Compression for Learning Temporal Action Abstractions in Controlcited by 21
  12. 2023Is Imitation All You Need? Generalized Decision-Making with Dual-Phase Trainingcited by 18
  13. 2024P3-PO: Prescriptive Point Priors for Visuo-Spatial Generalization of Robot Policiescited by 16
  14. 2024Premier-TACO is a Few-Shot Policy Learner: Pretraining Multitask Representation via Temporal Action-Driven Contrastive Losscited by 10
  15. 2024ARDuP: Active Region Video Diffusion for Universal Policiescited by 10
  16. 2017Structured Attentions for Visual Question Answeringcited by 6
  17. 2024What is Point Supervision Worth in Video Instance Segmentation?cited by 5
  18. 2024UVIS: Unsupervised Video Instance Segmentationcited by 5
  19. 2020Confidence-aware Adversarial Learning for Self-supervised Semantic Matchingcited by 3
  20. 2024WayEx: Waypoint Exploration using a Single Demonstrationcited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 5.

7 papers.

  1. 2025Trokens: Semantic-Aware Relational Trajectory Tokens for Few-Shot Action Recognitioncited by 7
  2. 2026Evaluating Feature Dependent Noise in Preference-based Reinforcement Learningcited by 2
  3. 2026RoDiF: Robust Direct Fine-Tuning of Diffusion Policies with Corrupted Human Feedbackcited by 2
  4. 2026Balancing Multiple Objectives in Urban Traffic Control with Reinforcement Learning from AI Feedbackcited by 1
  5. 2025PB2: Preference Space Exploration via Population-Based Methods in Preference-Based Reinforcement Learningcited by 0
  6. 2026Trust, Don't Trust, or Flip: Robust Preference-Based Reinforcement Learning with Multi-Expert Feedbackcited by 0
  7. 2026Selective Timestep Weighting and Advantage-Based Replay for Sample-Efficient Diffusion RLHFcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Search-Based Credit Assignment for Offline Preference-Based Reinforcement Learning
  2. 2021Towards Sample-efficient Apprenticeship Learning from Suboptimal Demonstration
  3. 2024Multi-Type Preference Learning: Empowering Preference-Based Reinforcement Learning with Equal Preferences
  4. 2024PrefCLM: Enhancing Preference-based Reinforcement Learning with Crowdsourced Large Language Models ↗
  5. 2025LAPP: Large Language Model Feedback for Preference-Driven Reinforcement Learning
  6. 2025Efficiently Generating Expressive Quadruped Behaviors via Language-Guided Preference Learning
  7. 2024Sample-Efficient Preference-based Reinforcement Learning with Dynamics Aware Rewards
  8. 2026Freeform Preference Learning for Robotic Manipulation