mlsanityData to 14 Oct 2026

PHIRL: Aligning Learned Rewards with Task Progress for Inverse Reinforcement Learning

arXiv 2609.31855 ↗First posted 25 Sep 2026cs.RO cs.AIcited by 0

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

also in 1 more
Abstract (arXiv)

Human demonstrations provide dense policy-level information but sometimes lack local precision. Human feedback presents accurate local critiques, but offers sparse evaluations rather than direct policy guidance. We propose Progress-Heuristicized Inverse Reinforcement Learning (PHIRL), a data-efficient framework that learns robust reward functions by jointly leveraging demonstrations and feedback. Specifically, we use progress, a feedback modality that describes cumulative task completion. PHIRL iteratively infers a reward function from demonstrations via inverse reinforcement learning, calculates the learned rewards over the progress-annotated demonstrations, and aligns the rewards with progress annotations over four dimensions. We evaluate PHIRL on real and simulated robot tasks, with additional exploration using a fine-tuned vision-language model to provide progress feedback. Results demonstrate that PHIRL significantly outperforms the baselines, achieving substantially higher environmental return rewards and task success with only twenty percent of demonstrations annotated. Analysis of reward-hacking scenarios demonstrates that PHIRL learned reward functions are reliable against exploitation.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 48 of its 55 references that are in the map. First posted on arXiv 25 Sep 2026.

Builds on

48 of 55 references.

Closest ancestors · 11

Same direction, newest first

  1. 2026TOPReward: Token Probabilities as Hidden Zero-Shot Rewards for Robotics22 Feb
  2. 2025Inverse Reinforcement Learning Meets Large Language Model Post-Training: Basics, Advances, and Opportunities17 Jul
  3. 2025CHARM: Considering Human Attributes for Reinforcement Modeling16 Jun
  4. 2024How Much Progress Did I Make? An Unexplored Human Feedback Signal for Teaching Robots8 Jul
  5. 2024Learning Reward and Policy Jointly from Demonstration and Preference Improves Alignment11 Jun
  6. 2024A Unified Linear Programming Framework for Offline Reward Learning from Human Demonstrations and Feedback20 May
  7. 2024Improving Reinforcement Learning from Human Feedback with Efficient Reward Model Ensemble30 Jan
  8. 2023From "Thumbs Up" to "10 out of 10": Reconsidering Scalar Feedback in Interactive Reinforcement Learning17 Nov
  9. 2020The EMPATHIC Framework for Task Learning from Implicit Human Feedback28 Sep
  10. 2018Reward learning from human preferences and demonstrations in Atari15 Nov
  11. 2017Deep reinforcement learning from human preferences12 Jun

Other references · 37

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  3. 2022Generative Adversarial Networkscited by 5,627
  4. 2023Diffusion Policy: Visuomotor Policy Learning via Action Diffusioncited by 2,838
  5. 2019Optuna: A Next-generation Hyperparameter Optimization Frameworkcited by 2,501
  6. 2025Gemma 3 Technical Reportcited by 1,824
  7. 2018DeepMind Control Suitecited by 956
  8. 2021What Matters in Learning from Offline Human Demonstrations for Robot Manipulationcited by 725
  9. 2020robosuite: A Modular Simulation Framework and Benchmark for Robot Learningcited by 511
  10. 2017Learning Robust Rewards with Adversarial Inverse Reinforcement Learningcited by 490
  11. 2025UniVLA: Learning to Act Anywhere with Task-centric Latent Actionscited by 410
  12. 2024Universal Manipulation Interface: In-The-Wild Robot Teaching Without In-The-Wild Robotscited by 394
25 more
  1. 2025Rubrics as Rewards: Reinforcement Learning Beyond Verifiable Domainscited by 366
  2. 2024EgoMimic: Scaling Imitation Learning via Egocentric Videocited by 175
  3. 2025OpenRubrics: Towards Scalable Synthetic Rubric Generation for Reward Modeling and LLM Alignmentcited by 132
  4. 2024Vision Language Models are In-Context Value Learnerscited by 90
  5. 2020Safe Imitation Learning via Fast Bayesian Reward Inference from Preferencescited by 73
  6. 2019Learning Reward Functions by Integrating Human Demonstrations and Preferencescited by 64
  7. 2025ReWiND: Language-Guided Rewards Teach Robot Policies without New Demonstrationscited by 60
  8. 2026Mind the Sim2Real Gap in User Simulation for Agentic Taskscited by 54
  9. 2021A Survey on Interactive Reinforcement Learning: Design Principles and Open Challengescited by 41
  10. 2024Comprehensive Overview of Reward Engineering and Shaping in Advancing Reinforcement Learning Applicationscited by 38
  11. 2024Hybrid Inverse Reinforcement Learningcited by 32
  12. 2024Rank2Reward: Learning Shaped Reward Functions from Passive Videocited by 32
  13. 2023Thought Cloning: Learning to Think while Acting by Imitating Human Thinkingcited by 30
  14. 2022Unified Learning from Demonstrations, Corrections, and Preferences during Physical Human-Robot Interactioncited by 19
  15. 2026PROGRESSLM: Towards Progress Reasoning in Vision-Language Modelscited by 15
  16. 2024PROGRESSOR: A Perceptually Guided Reward Estimator with Self-Supervised Online Refinementcited by 13
  17. 2026chi0: Resource-Aware Robust Manipulation via Taming Distributional Inconsistenciescited by 12
  18. 2023LgTS: Dynamic Task Sampling using LLM-generated sub-goals for Reinforcement Learning Agentscited by 11
  19. 2019Better-than-Demonstrator Imitation Learning via Automatically-Ranked Demonstrationscited by 6
  20. 2022Learning Multi-Task Transferable Rewards via Variational Inverse Reinforcement Learningcited by 5
  21. 2025FLEX: A Framework for Learning Robot-Agnostic Force-based Skills Involving Sustained Contact Object Manipulationcited by 2
  22. 2025Demonstration Sidetracks: Categorizing Systematic Non-Optimality in Human Demonstrationscited by 2
  23. 2026Learning from Demonstrations via Capability-Aware Goal Samplingcited by 1
  24. 2026Reward Transfer from Inverse Reinforcement Learning: A Coupled Minimax Approachcited by 1
  25. 2026Step-Level Preference Learning for Generative Agents in Social Simulationscited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2023BC-IRL: Learning Generalizable Reward Functions from Demonstrations
  2. 2023Distance-rank Aware Sequential Reward Learning for Inverse Reinforcement Learning with Sub-optimal Demonstrations
  3. 2026Reward-DAgger: Robot-Gated Interactive Imitation Learning with General-Purpose Progress-Based Reward Models
  4. 2026From Demonstrations to Rewards: Test-Time Prompt Optimization for VLM Reward Models
  5. 2019From Language to Goals: Inverse Reinforcement Learning for Vision-Based Instruction Following
  6. 2026Reward-Conditioned Reinforcement Learning
  7. 2026Inverse RL Helps Align AI by Imitating Humans
  8. 2020Feature Expansive Reward Learning: Rethinking Human Input