# Learning Personalized Driving Styles via Reinforcement Learning from Human Feedback

arXiv: https://arxiv.org/abs/2503.10434 · HTML: https://mlsanity.com/paper/2503.10434/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- Placed in Imitation-learning end-to-end planners, one of 9 directions in End-to-end autonomous driving.
- Builds on 35 of its 43 references that are in the map.
- First posted on arXiv 13 Mar 2025.

## Placed in

- End-to-end autonomous driving / [Imitation-learning end-to-end planners](https://mlsanity.com/driving/imitation-learning-end-to-end-planners/)
- LLM alignment / RLHF / preference optimisation / [Reward modeling and RLHF](https://mlsanity.com/alignment/reward-modeling-and-rlhf/)
- Deep reinforcement learning methods / [RL for autonomous driving and traffic](https://mlsanity.com/deeprl/rl-for-autonomous-driving-and-traffic/)

## Builds on (most cited first, 30 of 35)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning](https://mlsanity.com/paper/2501.12948/) (2025)
- [Deep reinforcement learning from human preferences](https://mlsanity.com/paper/1706.03741/) (2017)
- [Planning-oriented Autonomous Driving](https://mlsanity.com/paper/2212.10156/) (2022)
- [Training Diffusion Models with Reinforcement Learning](https://mlsanity.com/paper/2305.13301/) (2023)
- [Diffusion Model Alignment Using Direct Preference Optimization](https://mlsanity.com/paper/2311.12908/) (2023)
- [Social GAN: Socially Acceptable Trajectories with Generative Adversarial Networks](https://mlsanity.com/paper/1803.10892/) (2018)
- [VAD: Vectorized Scene Representation for Efficient Autonomous Driving](https://mlsanity.com/paper/2303.12077/) (2023)
- [End-to-end Driving via Conditional Imitation Learning](https://mlsanity.com/paper/1710.02410/) (2017)
- [TransFuser: Imitation with Transformer-Based Sensor Fusion for Autonomous Driving](https://mlsanity.com/paper/2205.15997/) (2022)
- [RLHF-V: Towards Trustworthy MLLMs via Behavior Alignment from Fine-grained Correctional Human Feedback](https://mlsanity.com/paper/2312.00849/) (2023)
- [NAVSIM: Data-Driven Non-Reactive Autonomous Vehicle Simulation and Benchmarking](https://mlsanity.com/paper/2406.15349/) (2024)
- [TNT: Target-driveN Trajectory Prediction](https://mlsanity.com/paper/2008.08294/) (2020)
- [DiffusionDrive: Truncated Diffusion Model for End-to-End Autonomous Driving](https://mlsanity.com/paper/2411.15139/) (2024)
- [DenseTNT: End-to-end Trajectory Prediction from Dense Goal Sets](https://mlsanity.com/paper/2108.09640/) (2021)
- [VADv2: End-to-End Vectorized Autonomous Driving via Probabilistic Planning](https://mlsanity.com/paper/2402.13243/) (2024)
- [Diffusion Policy Policy Optimization](https://mlsanity.com/paper/2409.00588/) (2024)
- [Hydra-MDP: End-to-end Multimodal Planning with Multi-target Hydra-Distillation](https://mlsanity.com/paper/2406.06978/) (2024)
- [MultiPath++: Efficient Information Fusion and Trajectory Aggregation for Behavior Prediction](https://mlsanity.com/paper/2111.14973/) (2021)
- [MotionDiffuser: Controllable Multi-Agent Motion Prediction using Diffusion](https://mlsanity.com/paper/2306.03083/) (2023)
- [Scene Transformer: A unified architecture for predicting multiple agent trajectories](https://mlsanity.com/paper/2106.08417/) (2021)
- [GoalFlow: Goal-Driven Flow Matching for Multimodal Trajectories Generation in End-to-End Autonomous Driving](https://mlsanity.com/paper/2503.05689/) (2025)
- [ViP3D: End-to-end Visual Trajectory Prediction via 3D Agent Queries](https://mlsanity.com/paper/2208.01582/) (2022)
- [GRAPE: Generalizing Robot Policy via Preference Alignment](https://mlsanity.com/paper/2411.19309/) (2024)
- [SocialVAE: Human Trajectory Prediction using Timewise Latents](https://mlsanity.com/paper/2203.08207/) (2022)

## Built on by (most cited first, 17 of 17)

- [Generalized Trajectory Scoring for End-to-end Multimodal Planning](https://mlsanity.com/paper/2506.06664/) (2025)
- [Driving on Registers](https://mlsanity.com/paper/2601.05083/) (2026)
- [DriveDPO: Policy Learning via Safety DPO For End-to-End Autonomous Driving](https://mlsanity.com/paper/2509.17940/) (2025)
- [Plan-R1: Safe and Feasible Trajectory Planning as Language Modeling](https://mlsanity.com/paper/2505.17659/) (2025)
- [EvaDrive: Evolutionary Adversarial Policy Optimization for End-to-End Autonomous Driving](https://mlsanity.com/paper/2508.09158/) (2025)
- [Test-Time Trajectory Optimization for Autonomous Driving](https://mlsanity.com/paper/2606.07170/) (2026)
- [TransDiffuser: Diverse Trajectory Generation with Decorrelated Multi-modal Representation for End-to-end Autonomous Driving](https://mlsanity.com/paper/2505.09315/) (2025)
- [WPT: World-to-Policy Transfer via Online World Model Distillation](https://mlsanity.com/paper/2511.20095/) (2025)
- [Can Users Specify Driving Speed? Bench2Drive-Speed: Benchmark and Baselines for Desired-Speed Conditioned Autonomous Driving](https://mlsanity.com/paper/2603.25672/) (2026)
- [DriveMA: Driving Vision-Language-Action Models with verifiable Meta-Actions](https://mlsanity.com/paper/2605.31271/) (2026)
- [PerlAD: Towards Enhanced Closed-loop End-to-end Autonomous Driving with Pseudo-simulation-based Reinforcement Learning](https://mlsanity.com/paper/2603.14908/) (2026)
- [Causality-Aware End-to-End Autonomous Driving via Ego-Centric Joint Scene Modeling](https://mlsanity.com/paper/2605.13646/) (2026)
- [DriveMA: Rethinking Language Interfaces in Driving VLAs with One-Step Meta-Actions](https://mlsanity.com/paper/2605.21273/) (2026)
- [UniTeD: Unified Temporal Diffusion for Joint Perception and Planning in Autonomous Driving](https://mlsanity.com/paper/2606.25736/) (2026)
- [CoIRL-AD: Collaborative-Competitive Imitation-Reinforcement Learning in Latent World Models for Autonomous Driving](https://mlsanity.com/paper/2510.12560/) (2025)
- [PersonaDrive: Controllable Trajectory Prediction with Multi-Dimensional Driving Personas](https://mlsanity.com/paper/2608.15230/) (2026)
- [YOUDrive: Driving Style as a Composable Direction for End-to-End Autonomous Driving](https://mlsanity.com/paper/2610.14802/) (2026)

## Abstract (arXiv)

Generating human-like and adaptive trajectories is essential for autonomous driving in dynamic environments. While generative models have shown promise in synthesizing feasible trajectories, they often fail to capture the nuanced variability of personalized driving styles due to dataset biases and distributional shifts. To address this, we introduce TrajHF, a human feedback-driven finetuning framework for generative trajectory models, designed to align motion planning with diverse driving styles. TrajHF incorporates multi-conditional denoiser and reinforcement learning with human feedback to refine multi-modal trajectory generation beyond conventional imitation learning. This enables better alignment with human driving preferences while maintaining safety and feasibility constraints. TrajHF achieves performance comparable to the state-of-the-art on NavSim benchmark. TrajHF sets a new paradigm for personalized and adaptable trajectory generation in autonomous driving.