mlsanityData to 14 Oct 2026

FLoRA: Sample-Efficient Preference-based RL via Low-Rank Style Adaptation of Reward Functions

arXiv 2504.10002 ↗First posted 14 Apr 2025cs.RO cs.LGcited by 3

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

also in 1 more
Abstract (arXiv)

Preference-based reinforcement learning (PbRL) is a suitable approach for style adaptation of pre-trained robotic behavior: adapting the robot's policy to follow human user preferences while still being able to perform the original task. However, collecting preferences for the adaptation process in robotics is often challenging and time-consuming. In this work we explore the adaptation of pre-trained robots in the low-preference-data regime. We show that, in this regime, recent adaptation approaches suffer from catastrophic reward forgetting (CRF), where the updated reward model overfits to the new preferences, leading the agent to become unable to perform the original task. To mitigate CRF, we propose to enhance the original reward model with a small number of parameters (low-rank matrices) responsible for modeling the preference adaptation. Our evaluation shows that our method can efficiently and effectively adjust robotic behavior to human preferences across simulation benchmark tasks and multiple real-world robotic tasks.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 47 of its 57 references that are in the map. First posted on arXiv 14 Apr 2025.

Builds on

47 of 57 references.

Closest ancestors · 11

Same direction, newest first

  1. 2024PREDILECT: Preferences Delineated with Zero-Shot Language-based Reasoning in Reinforcement Learning23 Feb
  2. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback27 Jul
  3. 2023On The Fragility of Learned Reward Functions9 Jan
  4. 2022Scaling Laws for Reward Model Overoptimization19 Oct
  5. 2022Reward Uncertainty for Exploration in Preference-based Reinforcement Learning24 May
  6. 2022SURF: Semi-supervised Reward Learning with Data Augmentation for Feedback-efficient Preference-based Reinforcement Learning18 Mar
  7. 2022Uncertainty Estimation for Language Reward Models14 Mar
  8. 2021PEBBLE: Feedback-Efficient Interactive Reinforcement Learning via Relabeling Experience and Unsupervised Pre-training9 Jun
  9. 2019Fine-Tuning Language Models from Human Preferences18 Sep
  10. 2018Reward learning from human preferences and demonstrations in Atari15 Nov
  11. 2017Deep reinforcement learning from human preferences12 Jun

Other references · 36

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  5. 2018A Style-Based Generator Architecture for Generative Adversarial Networkscited by 6,013
  6. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  7. 2021On the Opportunities and Risks of Foundation Modelscited by 4,033
  8. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,864
  9. 2016Concrete Problems in AI Safetycited by 1,642
  10. 2024Octo: An Open-Source Generalist Robot Policycited by 1,484
  11. 2019Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learningcited by 1,204
  12. 2023Open X-Embodiment: Robotic Learning Datasets and RT-X Modelscited by 862
24 more
  1. 2020Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuningcited by 671
  2. 2022Improving alignment of dialogue agents via targeted human judgementscited by 531
  3. 2020dm_control: Software and Tasks for Continuous Controlcited by 375
  4. 2018Measuring the Intrinsic Dimension of Objective Landscapescited by 370
  5. 2018Quantifying Generalization in Reinforcement Learningcited by 352
  6. 2023Orbit: A Unified Simulation Framework for Interactive Robot Learning Environmentscited by 295
  7. 2019Causal Confusion in Imitation Learningcited by 288
  8. 2017Inverse Reward Designcited by 249
  9. 2024Reinforcement learningcited by 247
  10. 2023ZipLoRA: Any Subject in Any Style by Effectively Merging LoRAscited by 194
  11. 2023Aligning Language Models with Preferences through f-divergence Minimizationcited by 120
  12. 2019Observational Overfitting in Reinforcement Learningcited by 91
  13. 2018GPU-Accelerated Robotic Simulation for Distributed Reinforcement Learningcited by 90
  14. 2018A Dissection of Overfitting and Generalization in Continuous Reinforcement Learningcited by 87
  15. 2022Few-Shot Preference Learning for Human-in-the-Loop RLcited by 87
  16. 2024AutoRT: Embodied Foundation Models for Large Scale Orchestration of Robotic Agentscited by 87
  17. 2024Multi-LoRA Composition for Image Generationcited by 80
  18. 2023Preference Transformer: Modeling Human Preferences using Transformers for RLcited by 79
  19. 2020Quantifying Differences in Reward Functionscited by 63
  20. 2024Plan-Seq-Learn: Language Model Guided RL for Solving Long Horizon Robotics Taskscited by 52
  21. 2022Invariance in Policy Optimisation and Partial Identifiability in Reward Learningcited by 46
  22. 2017Repeated Inverse Reinforcement Learningcited by 29
  23. 2022Explaining Preferences with Shapley Valuescited by 6
  24. 2020An Empowerment-based Solution to Robotic Manipulation Tasks with Sparse Rewardscited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 1.

3 papers.

  1. 2025MultLFG: Training-free Multi-LoRA composition using Frequency-domain Guidancecited by 6
  2. 2025Self-correcting Reward Shaping via Language Models for Reinforcement Learning Agents in Gamescited by 1
  3. 2026Augmenting Game AI with Deep Reinforcement Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Sample-Efficient Preference-based Reinforcement Learning with Dynamics Aware Rewards
  2. 2025Residual Reward Models: Leveraging Prior Knowledge for Efficient Preference-based Reinforcement Learning in Robotics
  3. 2023Inverse Preference Learning: Preference-based RL without a Reward Function
  4. 2023Direct Preference-based Policy Optimization without Reward Modeling
  5. 2024Multi-Type Preference Learning: Empowering Preference-Based Reinforcement Learning with Equal Preferences
  6. 2023Provable Reward-Agnostic Preference-Based Reinforcement Learning
  7. 2024RIME: Robust Preference-based Reinforcement Learning with Noisy Preferences
  8. 2024PrefCLM: Enhancing Preference-based Reinforcement Learning with Crowdsourced Large Language Models ↗