mlsanityData to 7 Oct 2026

Reinforcement Learning with Human Feedback for Realistic Traffic Simulation

arXiv 2309.00709 ↗First posted 1 Sep 2023cs.AI cs.LG cs.ROcited by 20

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

In light of the challenges and costs of real-world testing, autonomous vehicle developers often rely on testing in simulation for the creation of reliable systems. A key element of effective simulation is the incorporation of realistic traffic models that align with human knowledge, an aspect that has proven challenging due to the need to balance realism and diversity. This works aims to address this by developing a framework that employs reinforcement learning with human preference (RLHF) to enhance the realism of existing traffic models. This study also identifies two main challenges: capturing the nuances of human preferences on realism and the unification of diverse traffic simulation models. To tackle these issues, we propose using human feedback for alignment and employ RLHF due to its sample efficiency. We also introduce the first dataset for realism alignment in traffic modeling to support such research. Our framework, named TrafficRLHF, demonstrates its proficiency in generating realistic traffic scenarios that are well-aligned with human preferences, as corroborated by comprehensive evaluations on the nuScenes dataset.

Key facts. Cited by 20 papers in the map (data through 7 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 22 of its 34 references that are in the map. First posted on arXiv 1 Sep 2023.

Builds on

22 of 34 references.

Closest ancestors · 4

Same direction, newest first

  1. 2022Improving Multimodal Interactive Agents with Reinforcement Learning from Human Feedback21 Nov
  2. 2022Scaling Laws for Reward Model Overoptimization19 Oct
  3. 2019Fine-Tuning Language Models from Human Preferences18 Sep
  4. 2017Deep reinforcement learning from human preferences12 Jun

Other references · 18

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2019nuScenes: A multimodal dataset for autonomous drivingcited by 4,485
  3. 2020Learning to summarize from human feedbackcited by 2,562
  4. 2023Training Diffusion Models with Reinforcement Learningcited by 939
  5. 2019MultiPath: Multiple Probabilistic Anchor Trajectory Hypotheses for Behavior Predictioncited by 417
  6. 2023ChatGPT for Robotics: Design Principles and Model Abilitiescited by 296
  7. 2019Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialogcited by 283
  8. 2021A Survey of Human-in-the-loop for Machine Learningcited by 214
  9. 2021TrafficSim: Learning to Simulate Realistic Multi-Agent Behaviorscited by 192
  10. 2022Guided Conditional Diffusion for Controllable Traffic Simulationcited by 189
  11. 2021Generating Useful Accident-Prone Driving Scenarios via a Learned Traffic Priorcited by 136
  12. 2021AdvSim: Generating Safety-Critical Scenarios for Self-Driving Vehiclescited by 132
6 more
  1. 2022TrafficGen: Learning to Generate Diverse and Realistic Traffic Scenarioscited by 104
  2. 2021Colossal-AI: A Unified Deep Learning System For Large-Scale Parallel Trainingcited by 96
  3. 2022BITS: Bi-level Imitation for Traffic Simulationcited by 80
  4. 2022ScePT: Scene-consistent, Policy-based Trajectory Predictions for Planningcited by 63
  5. 2022AdvDO: Realistic Adversarial Attacks for Trajectory Predictioncited by 48
  6. 2020Adversarial Evaluation of Autonomous Vehicles in Lane-Change Scenarioscited by 42

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 7; 2025: 8; 2026: 4.

20 papers.

Sort:

By area:

  1. 2024Gen-Drive: Enhancing Diffusion Generative Driving Policies with Reward Modeling and Reinforcement Learning Fine-tuningcited by 41
  2. 2024Closed-Loop Supervised Fine-Tuning of Tokenized Traffic Modelscited by 40
  3. 2023Data-driven Traffic Simulation: A Comprehensive Reviewcited by 29
  4. 2024Learning to Drive via Asymmetric Self-Playcited by 11
  5. 2024Text-to-Drive: Diverse Driving Behavior Synthesis via Large Language Modelscited by 8
  6. 2025DriveGen: Towards Infinite Diverse Traffic Scenarios with Large Modelscited by 7
  7. 2025SPACeR: Self-Play Anchoring with Centralized Reference Modelscited by 7
  8. 2025Direct Post-Training Preference Alignment for Multi-Agent Motion Generation Models Using Implicit Feedback from Pre-training Demonstrationscited by 6
  9. 2025RIFT: Group-Relative RL Fine-Tuning for Realistic and Controllable Traffic Simulationcited by 4
  10. 2024Adversarial and Reactive Traffic Entities for Behavior-Realistic Driving Simulation: A Reviewcited by 3
  11. 2025Direct Preference Optimization-Enhanced Multi-Guided Diffusion Model for Traffic Scenario Generationcited by 2
  12. 2025SceneStreamer: Continuous Scenario Generation as Next Token Group Predictioncited by 2
8 more
  1. 2026RLFTSim: Realistic and Controllable Multi-Agent Traffic Simulation via Reinforcement Learning Fine-Tuningcited by 2
  2. 2024A Deep Generative Framework for Joint Households and Individuals Population Synthesiscited by 1
  3. 2024AI-Powered CPS-Enabled Vulnerable-User-Aware Urban Transportation Digital Twin: Methods and Applicationscited by 1
  4. 2025Provably Mitigating Corruption, Overoptimization, and Verbosity Simultaneously in Offline and Online RLHF/DPO Alignmentcited by 1
  5. 2026World Models as Adversaries: Multi-Agent Self-Play Fine-Tuning for Robust Motion Planningcited by 1
  6. 2025Realistic Urban Traffic Generator using Decentralized Federated Learning for the SUMO simulatorcited by 0
  7. 2026Controllable Sim Agents with Behavior Latentscited by 0
  8. 2026RENEW: Towards Learning World Models and Repairing Model Exploitation from Preferencescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Optimizing Autonomous Driving for Safety: A Human-Centric Approach with LLM-Enhanced RLHF
  2. 2024Trustworthy Human-AI Collaboration: Reinforcement Learning with Human Feedback and Physics Knowledge for Safe Autonomous Driving
  3. 2023Learning Realistic Traffic Agents in Closed-loop
  4. 2024Human-compatible driving partners through data-regularized self-play reinforcement learning
  5. 2025Impact of Collective Behaviors of Autonomous Vehicles on Urban Traffic Dynamics: A Multi-Agent Reinforcement Learning Approach
  6. 2025Advancing Multi-agent Traffic Simulation via R1-Style Reinforcement Fine-Tuning
  7. 2022Driver Dojo: A Benchmark for Generalizable Reinforcement Learning for Autonomous Driving
  8. 2023A Survey of Reinforcement Learning from Human Feedback