mlsanityData to 7 Oct 2026

Adversarial Policy Optimization for Offline Preference-based Reinforcement Learning

arXiv 2503.05306 ↗First posted 7 Mar 2025cs.LG cs.AIcited by 2

Not on the map. Why?

Abstract (arXiv)

In this paper, we study offline preference-based reinforcement learning (PbRL), where learning is based on pre-collected preference feedback over pairs of trajectories. While offline PbRL has demonstrated remarkable empirical success, existing theoretical approaches face challenges in ensuring conservatism under uncertainty, requiring computationally intractable confidence set constructions. We address this limitation by proposing Adversarial Preference-based Policy Optimization (APPO), a computationally efficient algorithm for offline PbRL that guarantees sample complexity bounds without relying on explicit confidence sets. By framing PbRL as a two-player game between a policy and a model, our approach enforces conservatism in a tractable manner. Using standard assumptions on function approximation and bounded trajectory concentrability, we derive a sample complexity bound. To our knowledge, APPO is the first offline PbRL algorithm to offer both statistical efficiency and practical applicability. Experimental results on continuous control tasks demonstrate that APPO effectively learns from complex datasets, showing comparable performance with existing state-of-the-art methods.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 53 of its 67 references that are in the map. First posted on arXiv 7 Mar 2025.

Builds on

53 of 67 references.

References · 53

  1. 2017Attention Is All You Needcited by 59,932
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2017Proximal Policy Optimization Algorithmscited by 14,441
  4. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  5. 2017Deep reinforcement learning from human preferencescited by 3,846
  6. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  7. 2013Playing Atari with Deep Reinforcement Learningcited by 3,211
  8. 2020Learning to summarize from human feedbackcited by 2,562
  9. 2015Trust Region Policy Optimizationcited by 2,503
  10. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  11. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  12. 2022RT-1: Robotics Transformer for Real-World Control at Scalecited by 2,007
38 more
  1. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,722
  2. 2021Offline Reinforcement Learning with Implicit Q-Learningcited by 1,245
  3. 2019Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learningcited by 1,168
  4. 2018QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulationcited by 730
  5. 2018Reward learning from human preferences and demonstrations in Ataricited by 314
  6. 2021PEBBLE: Feedback-Efficient Interactive Reinforcement Learning via Relabeling Experience and Unsupervised Pre-trainingcited by 298
  7. 2019Extrapolating Beyond Suboptimal Demonstrations via Inverse Reinforcement Learning from Observationscited by 256
  8. 2023Principled Reinforcement Learning with Human Feedback from Pairwise or K-wise Comparisonscited by 244
  9. 2019Provably Efficient Exploration in Policy Optimizationcited by 203
  10. 2017A unified view of entropy-regularized Markov decision processescited by 174
  11. 2021Pessimistic Model-based Offline Reinforcement Learning under Partial Coveragecited by 160
  12. 2022Hybrid RL: Using Both Offline and Online Data Can Make RL Efficientcited by 158
  13. 2017Interactive Learning from Policy-Dependent Human Feedbackcited by 155
  14. 2024A Minimaximalist Approach to Reinforcement Learning from Human Feedbackcited by 141
  15. 2017Deep TAMER: Interactive Agent Shaping in High-Dimensional State Spacescited by 122
  16. 2022RAMBO-RL: Robust Adversarial Model-Based Offline Reinforcement Learningcited by 115
  17. 2022Adversarially Trained Actor Critic for Offline Reinforcement Learningcited by 111
  18. 2021Dueling RL: Reinforcement Learning with Trajectory Preferencescited by 105
  19. 2022SURF: Semi-supervised Reward Learning with Data Augmentation for Feedback-efficient Preference-based Reinforcement Learningcited by 96
  20. 2022Few-Shot Preference Learning for Human-in-the-Loop RLcited by 85
  21. 2022Human-in-the-loop: Provably Efficient Preference-based Reinforcement Learning with General Function Approximationcited by 80
  22. 2023Preference Transformer: Modeling Human Preferences using Transformers for RLcited by 77
  23. 2020A Game Theoretic Framework for Model Based Reinforcement Learningcited by 72
  24. 2023Inverse Preference Learning: Preference-based RL without a Reward Functioncited by 68
  25. 2020Preference-based Reinforcement Learning with Finite-Time Guaranteescited by 65
  26. 2019Dueling Posterior Sampling for Preference-Based Reinforcement Learningcited by 56
  27. 2023Benchmarks and Algorithms for Offline Preference-Based Reward Learningcited by 55
  28. 2023Provable Offline Preference-Based Reinforcement Learningcited by 55
  29. 2023Making RL with Preference-based Feedback Efficient via Randomizationcited by 46
  30. 2024Principled Penalty-based Methods for Bilevel Reinforcement Learning and RLHFcited by 41
  31. 2024Dataset Reset Policy Optimization for RLHFcited by 41
  32. 2023Direct Preference-based Policy Optimization without Reward Modelingcited by 36
  33. 2023Beyond Reward: Offline Preference-guided Policy Optimizationcited by 35
  34. 2023Adversarial Model for Offline Reinforcement Learningcited by 31
  35. 2023Provable Reward-Agnostic Preference-Based Reinforcement Learningcited by 25
  36. 2024Exploration-Driven Policy Optimization in RLHF: Theoretical Insights on Efficient Data Utilizationcited by 25
  37. 2023Survival Instinct in Offline Reinforcement Learningcited by 20
  38. 2024Listwise Reward Estimation for Offline Preference-based Reinforcement Learningcited by 14

50 most cited of 53 All 53 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026The Trajectory Alignment Coefficient in Two Acts: From Reward Tuning to Reward Learningcited by 0
  2. 2026From Reward-Free Representations to Preferences: Rethinking Offline Preference-Based Reinforcement Learningcited by 0