mlsanityData to 14 Oct 2026

Fine-tuning Behavioral Cloning Policies with Preference-Based Reinforcement Learning

arXiv 2509.26605 ↗First posted 30 Sep 2025cs.AI cs.LGcited by 1

On the map: Offline RL and imitation learning from data → Offline RL algorithms

also in 1 more
Abstract (arXiv)

Deploying reinforcement learning (RL) in robotics, industry, and health care is blocked by two obstacles: the difficulty of specifying accurate rewards and the risk of unsafe, data-hungry exploration. We address this by proposing a two-stage framework that first learns a safe initial policy from a reward-free dataset of expert demonstrations, then fine-tunes it online using preference-based human feedback. We provide the first principled analysis of this offline-to-online approach and introduce BRIDGE, a unified algorithm that integrates both signals via an uncertainty-weighted objective. We derive regret bounds that shrink with the number of offline demonstrations, explicitly connecting the quantity of offline data to online sample efficiency. We validate BRIDGE in discrete and continuous control MuJoCo environments, showing it achieves lower regret than both standalone behavioral cloning and online preference-based RL. Our work establishes a theoretical foundation for designing more sample-efficient interactive agents.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 8 of its 41 references that are in the map. First posted on arXiv 30 Sep 2025.

Builds on

8 of 41 references.

Closest ancestors · 1

Same direction, newest first

  1. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May

Other references · 7

  1. 2017Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrationscited by 715
  2. 2018Scalable agent alignment via reward modeling: a research directioncited by 447
  3. 2017Leveraging Demonstrations for Deep Reinforcement Learning on Robotics Problems with Sparse Rewardscited by 319
  4. 2019Challenges of Real-World Reinforcement Learningcited by 291
  5. 2025Zero-Shot Whole-Body Humanoid Control via Behavioral Foundation Modelscited by 56
  6. 2024Gymnasium: A Standard Interface for Reinforcement Learning Environmentscited by 46
  7. 2023Efficient Online Learning with Offline Datasets for Infinite Horizon MDPs: A Bayesian Approachcited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026MAVRL: Learning Reward Functions from Multiple Feedback Types with Amortized Variational Inferencecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Search-Based Credit Assignment for Offline Preference-Based Reinforcement Learning
  2. 2022Improving TD3-BC: Relaxed Policy Constraint for Offline Learning and Stable Online Fine-Tuning
  3. 2025Behavior Preference Regression for Offline Reinforcement Learning
  4. 2022Know Your Boundaries: The Necessity of Explicit Behavioral Cloning in Offline RL
  5. 2026Optimizing Neurorobot Policy under Limited Demonstration Data through Preference Regret
  6. 2026OHP-RL: Online Human Preference as Guidance in Reinforcement Learning for Robot Manipulation
  7. 2022Reward Uncertainty for Exploration in Preference-based Reinforcement Learning
  8. 2026When Life Gives You BC, Make Q-functions: Extracting Q-values from Behavior Cloning for On-Robot Reinforcement Learning