mlsanityData to 7 Oct 2026

TROFI: Trajectory-Ranked Offline Inverse Reinforcement Learning

arXiv 2506.22008 ↗First posted 27 Jun 2025cs.LG cs.AIcited by 0

On the map: Offline RL and imitation learning from data → Inverse reinforcement learning

Abstract (arXiv)

In offline reinforcement learning, agents are trained using only a fixed set of stored transitions derived from a source policy. However, this requires that the dataset be labeled by a reward function. In applied settings such as video game development, the availability of the reward function is not always guaranteed. This paper proposes Trajectory-Ranked OFfline Inverse reinforcement learning (TROFI), a novel approach to effectively learn a policy offline without a pre-defined reward function. TROFI first learns a reward function from human preferences, which it then uses to label the original dataset making it usable for training the policy. In contrast to other approaches, our method does not require optimal trajectories. Through experiments on the D4RL benchmark we demonstrate that TROFI consistently outperforms baselines and performs comparably to using the ground truth reward to learn policies. Additionally, we validate the efficacy of our method in a 3D game environment. Our studies of the reward model highlight the importance of the reward function in this setting: we show that to ensure the alignment of a value function to the actual future discounted reward, it is fundamental to have a well-engineered and easy-to-learn reward function.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Inverse reinforcement learning, one of 13 directions in Offline RL and imitation learning from data. Builds on 29 of its 39 references that are in the map. First posted on arXiv 27 Jun 2025.

Builds on

29 of 39 references.

Closest ancestors · 2

Same direction, newest first

  1. 2020Demonstration-efficient Inverse Reinforcement Learning in Procedurally Generated Environments4 Dec
  2. 2019Extrapolating Beyond Suboptimal Demonstrations via Inverse Reinforcement Learning from Observations12 Apr

Other references · 27

  1. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  2. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  3. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,777
  4. 2016Generative Adversarial Imitation Learningcited by 1,576
  5. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,406
  6. 2021Offline Reinforcement Learning with Implicit Q-Learningcited by 1,245
  7. 2021A Minimalist Approach to Offline Reinforcement Learningcited by 882
  8. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 750
  9. 2023Imitating Human Behaviour with Diffusion Modelscited by 260
  10. 2021Offline Reinforcement Learning with Fisher Divergence Critic Regularizationcited by 237
  11. 2022Discriminator-Weighted Offline Imitation Learning from Suboptimal Demonstrationscited by 89
  12. 2020Offline Learning from Demonstrations and Unlabeled Experiencecited by 72
15 more
  1. 2023Inverse Preference Learning: Preference-based RL without a Reward Functioncited by 68
  2. 2022How to Leverage Unlabeled Data in Offline Reinforcement Learningcited by 61
  3. 2023Optimal Transport for Offline Imitation Learningcited by 42
  4. 2023Direct Preference-based Policy Optimization without Reward Modelingcited by 36
  5. 2023Beyond Reward: Offline Preference-guided Policy Optimizationcited by 35
  6. 2020Semi-supervised reward learning for offline reinforcement learningcited by 22
  7. 2023Survival Instinct in Offline Reinforcement Learningcited by 20
  8. 2023MAHALO: Unifying Offline Reinforcement Learning and Imitation Learning from Observationscited by 18
  9. 2020"It's Unwieldy and It Takes a Lot of Time." Challenges and Opportunities for Creating Agents in Commercial Gamescited by 17
  10. 2019Winning Isn't Everything: Enhancing Game Development with Intelligent Agentscited by 15
  11. 2022Towards Informed Design and Validation Assistance in Computer Games Using Imitation Learningcited by 13
  12. 2023Technical Challenges of Deploying Reinforcement Learning Agents for Game Testing in AAA Gamescited by 12
  13. 2023The Provable Benefits of Unsupervised Data Sharing for Offline Reinforcement Learningcited by 10
  14. 2024Efficient Active Imitation Learning with Random Network Distillationcited by 7
  15. 2025Real-Time Diffusion Policies for Games: Enhancing Consistency Policies with Q-Ensemblescited by 3

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2022Semi-Supervised Offline Reinforcement Learning with Action-Free Trajectories
  2. 2025OffSim: Offline Simulator for Model-based Offline Inverse Reinforcement Learning
  3. 2024TrajDeleter: Enabling Trajectory Forgetting in Offline Reinforcement Learning Agents
  4. 2023CLARE: Conservative Model-Based Reward Learning for Offline Inverse Reinforcement Learning
  5. 2023Distance-rank Aware Sequential Reward Learning for Inverse Reinforcement Learning with Sub-optimal Demonstrations
  6. 2025Offline Reinforcement Learning via Inverse Optimization
  7. 2023Using Offline Data to Speed Up Reinforcement Learning in Procedurally Generated Environments
  8. 2023Beyond Uniform Sampling: Offline Reinforcement Learning with Imbalanced Datasets