Data to 7 Oct 2026

Preference Transformer: Modeling Human Preferences using Transformers for RL

arXiv 2303.00957 ↗First posted 2 Mar 2023cs.LG cs.AI cs.ROcited by 77

Not on the map. Why?

Cited by 44 in LLM alignment / RLHF / preference optimisation, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Preference-based reinforcement learning (RL) provides a framework to train agents using human preferences between two behaviors. However, preference-based RL has been challenging to scale since it requires a large amount of human feedback to learn a reward function aligned with human intent. In this paper, we present Preference Transformer, a neural architecture that models human preferences using transformers. Unlike prior approaches assuming human judgment is based on the Markovian rewards which contribute to the decision equally, we introduce a new preference model based on the weighted sum of non-Markovian rewards. We then design the proposed preference model using a transformer architecture that stacks causal and bidirectional self-attention layers. We demonstrate that Preference Transformer can solve a variety of control tasks using real human preferences, while prior approaches fail to work. We also show that Preference Transformer can induce a well-specified reward and attend to critical events in the trajectory by automatically capturing the temporal dependencies in human decision-making. Code is available on the project website: https://sites.google.com/view/preference-transformer.

Key facts. Cited by 77 papers in the map (data through 7 Oct 2026). Builds on 42 of its 69 references that are in the map. First posted on arXiv 2 Mar 2023.

Builds on

42 of 69 references.

References · 42

2017Attention Is All You Needcited by 59,932
2020Language Models are Few-Shot Learnerscited by 32,199
2022Training language models to follow instructions with human feedbackcited by 15,712
2016Layer Normalizationcited by 5,579
2021Zero-Shot Text-to-Image Generationcited by 4,276
2017Deep reinforcement learning from human preferencescited by 3,846
2020Learning to summarize from human feedbackcited by 2,562
2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,722
2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,528
2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,406
2021Offline Reinforcement Learning with Implicit Q-Learningcited by 1,245
2019Dota 2 with Large Scale Deep Reinforcement Learningcited by 1,084

30 more

2022A Generalist Agentcited by 802
2021What Matters in Learning from Offline Human Demonstrations for Robot Manipulationcited by 699
2020robosuite: A Modular Simulation Framework and Benchmark for Robot Learningcited by 484
2016Successor Features for Transfer in Reinforcement Learningcited by 331
2018Reward learning from human preferences and demonstrations in Ataricited by 314
2021PEBBLE: Feedback-Efficient Interactive Reinforcement Learning via Relabeling Experience and Unsupervised Pre-trainingcited by 298
2021Recursively Summarizing Books with Human Feedbackcited by 287
2019Stabilizing Transformers for Reinforcement Learningcited by 262
2017One-Shot Imitation Learningcited by 232
2020Learning Agile Robotic Locomotion Skills by Imitating Animalscited by 229
2018RUDDER: Return Decomposition for Delayed Rewardscited by 162
2020SUNRISE: A Simple Unified Framework for Ensemble Learning in Deep Reinforcement Learningcited by 150
2021Learning Language-Conditioned Robot Behavior from Offline Data and Crowd-Sourced Annotationcited by 117
2021B-Pref: Benchmarking Preference-Based Reinforcement Learningcited by 106
2022SURF: Semi-supervised Reward Learning with Data Augmentation for Feedback-efficient Preference-based Reinforcement Learningcited by 96
2022Few-Shot Preference Learning for Human-in-the-Loop RLcited by 85
2020Transformers for One-Shot Visual Imitationcited by 77
2020DisCor: Corrective Feedback in Reinforcement Learning via Distribution Correctioncited by 71
2019From Language to Goals: Inverse Reinforcement Learning for Vision-Based Instruction Followingcited by 67
2022Reward Uncertainty for Exploration in Preference-based Reinforcement Learningcited by 60
2022Models of human preference for learning reward functionscited by 54
2021Towards More Generalizable One-shot Visual Imitation Learningcited by 48
2021Learning Long-Term Reward Redistribution via Randomized Return Decompositioncited by 33
2020Learning Guidance Rewards with Trajectory-space Smoothingcited by 30
2019Evolving Rewards to Automate Reinforcement Learningcited by 28
2016Visual Closed-Loop Control for Pouring Liquidscited by 26
2016Collective Robot Reinforcement Learning with Distributed Asynchronous Guided Policy Searchcited by 25
2021Offline Preference-Based Apprenticeship Learningcited by 17
2022Non-Markovian Reward Modelling from Trajectory Labels via Interpretable Multiple Instance Learningcited by 13
2022Symbol Guided Hindsight Priors for Reward Learning from Human Preferencescited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 13; 2024: 28; 2025: 28; 2026: 8.

77 papers.

Sort:

By area:

2023Safe RLHF: Safe Reinforcement Learning from Human Feedbackcited by 630
2024Personalizing Reinforcement Learning from Human Feedback with Variational Preference Learningcited by 147
2023Policy Representation via Diffusion Probability Model for Reinforcement Learningcited by 105
2023Contrastive Preference Learning: Learning from Human Feedback without RLcited by 77
2023Inverse Preference Learning: Preference-based RL without a Reward Functioncited by 68
2024A Dense Reward View on Aligning Text-to-Image Diffusion with Preferencecited by 55
2024RIME: Robust Preference-based Reinforcement Learning with Noisy Preferencescited by 37
2025SafeDPO: A Simple Approach to Direct Preference Optimization with Enhanced Safetycited by 37
2023Beyond Reward: Offline Preference-guided Policy Optimizationcited by 35
2023Preference-grounded Token-level Guidance for Language Model Fine-tuningcited by 31
2025Reward Models in Deep Reinforcement Learning: A Surveycited by 24
2024Uni-RLHF: Universal Platform and Benchmark Suite for Reinforcement Learning with Diverse Human Feedbackcited by 20

38 more

2024Continuous Control with Coarse-to-fine Reinforcement Learningcited by 15
2023Learning to Discern: Imitating Heterogeneous Human Demonstrations with Preference and Representation Learningcited by 14
2024Listwise Reward Estimation for Offline Preference-based Reinforcement Learningcited by 14
2024Hindsight PRIORs for Reward Learning from Human Preferencescited by 12
2025Segmenting Text and Learning Their Rewards for Improved RLHF in Language Modelcited by 12
2023Transformers in Reinforcement Learning: A Surveycited by 11
2024Preference Alignment with Flow Matchingcited by 11
2023Guide Your Agent with Adaptive Multimodal Rewardscited by 10
2024Forward KL Regularized Preference Optimization for Aligning Diffusion Policiescited by 10
2023PEARL: Zero-shot Cross-task Preference Alignment and Robust Reward Learning for Robotic Manipulationcited by 9
2024Regularized Conditional Diffusion Model for Multi-Task Preference Alignmentcited by 9
2024Efficient Preference-based Reinforcement Learning via Aligned Experience Estimationcited by 9
2023Explore 3D Dance Generation via Reward Model from Automatically-Ranked Demonstrationscited by 8
2025SimulPL: Aligning Human Preferences in Simultaneous Machine Translationcited by 7
2025Contextual Online Uncertainty-Aware Preference Learning for Human Feedbackcited by 7
2025TREND: Tri-teaching for Robust Preference-based Reinforcement Learning with Demonstrationscited by 7
2026HISR: Hindsight Information Modulated Segmental Process Rewards For Multi-turn Agentic Reinforcement Learningcited by 7
2025Human Implicit Preference-Based Policy Fine-tuning for Multi-Agent Reinforcement Learning in USV Swarmcited by 5
2025Sentence-level Reward Model can Generalize Better for Aligning LLM from Human Preferencecited by 5
2024Improving Sample Efficiency of Reinforcement Learning with Background Knowledge from Large Language Modelscited by 4
2024S-EPOA: Overcoming the Indistinguishability of Segments with Skill-Driven Preference-Based Reinforcement Learningcited by 4
2024Offline Inverse Constrained Reinforcement Learning for Safe-Critical Decision Making in Healthcarecited by 4
2024Approximated Variational Bayesian Inverse Reinforcement Learning for Large Language Model Alignmentcited by 4
2025Influencing Humans to Conform to Preference Models for RLHFcited by 4
2025Policy Learning from Large Vision-Language Model Feedback without Reward Modelingcited by 4
2023Conditions on Preference Relations that Guarantee the Existence of Optimal Policiescited by 3
2024DynaMITE-RL: A Dynamic Model for Improved Temporal Meta-Reinforcement Learningcited by 3
2024Dishonesty in Helpful and Harmless Alignmentcited by 3
2024Preference Elicitation for Offline Reinforcement Learningcited by 3
2024Hindsight Preference Learning for Offline Preference-based Reinforcement Learningcited by 3
2025Sigmoid Self-Attention has Lower Sample Complexity than Softmax Self-Attention: A Mixture-of-Experts Perspectivecited by 3
2025FLoRA: Sample-Efficient Preference-based RL via Low-Rank Style Adaptation of Reward Functionscited by 3
2025MisoDICE: Multi-Agent Imitation from Unlabeled Mixed-Quality Demonstrationscited by 3
2025CLARIFY: Contrastive Preference Reinforcement Learning for Untangling Ambiguous Queriescited by 3
2025Aligning Humans and Robots via Reinforcement Learning from Implicit Human Feedbackcited by 3
2025STAIR: Addressing Stage Misalignment through Temporal-Aligned Preference Reinforcement Learningcited by 3
2024Multi-Type Preference Learning: Empowering Preference-Based Reinforcement Learning with Equal Preferencescited by 2
2024Evolutionary Retrofittingcited by 2

50 most cited of 77 All 77 (JSON)