mlsanityData to 14 Oct 2026

Transductive Reward Inference on Graph

arXiv 2402.03661 ↗First posted 6 Feb 2024cs.LG cs.AIcited by 0

On the map: Offline RL and imitation learning from data → Offline RL algorithms

also in 1 more
Abstract (arXiv)

In this study, we present a transductive inference approach on that reward information propagation graph, which enables the effective estimation of rewards for unlabelled data in offline reinforcement learning. Reward inference is the key to learning effective policies in practical scenarios, while direct environmental interactions are either too costly or unethical and the reward functions are rarely accessible, such as in healthcare and robotics. Our research focuses on developing a reward inference method based on the contextual properties of information propagation on graphs that capitalizes on a constrained number of human reward annotations to infer rewards for unlabelled data. We leverage both the available data and limited reward annotations to construct a reward propagation graph, wherein the edge weights incorporate various influential factors pertaining to the rewards. Subsequently, we employ the constructed graph for transductive reward inference, thereby estimating rewards for unlabelled data. Furthermore, we establish the existence of a fixed point during several iterations of the transductive inference process and demonstrate its at least convergence to a local optimum. Empirical evaluations on locomotion and robotic manipulation tasks validate the effectiveness of our approach. The application of our inferred rewards improves the performance in offline reinforcement learning tasks.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 33 of its 69 references that are in the map. First posted on arXiv 6 Feb 2024.

Builds on

33 of 69 references.

Closest ancestors · 8

Same direction, newest first

  1. 2022A Survey on Offline Reinforcement Learning: Taxonomy, Review, and Open Problems2 Mar
  2. 2022How to Leverage Unlabeled Data in Offline Reinforcement Learning3 Feb
  3. 2021Offline Reinforcement Learning with Fisher Divergence Critic Regularization14 Mar
  4. 2020Semi-supervised reward learning for offline reinforcement learning12 Dec
  5. 2020Critic Regularized Regression26 Jun
  6. 2020RL Unplugged: A Suite of Benchmarks for Offline Reinforcement Learning24 Jun
  7. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May
  8. 2020Keep Doing What Worked: Behavioral Modelling Priors for Offline Reinforcement Learning19 Feb

Other references · 25

  1. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  2. 2016Generative Adversarial Imitation Learningcited by 1,597
  3. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,241
  4. 2019Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learningcited by 1,204
  5. 2018DeepMind Control Suitecited by 956
  6. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 782
  7. 2017Learning Robust Rewards with Adversarial Inverse Reinforcement Learningcited by 490
  8. 2016Guided Cost Learning: Deep Inverse Optimal Control via Policy Optimizationcited by 406
  9. 2020dm_control: Software and Tasks for Continuous Controlcited by 375
  10. 2019Label Propagation for Deep Semi-supervised Learningcited by 289
  11. 2019RoboNet: Large-Scale Multi-Robot Learningcited by 270
  12. 2018Learning to Propagate Labels: Transductive Propagation Network for Few-shot Learningcited by 211
13 more
  1. 2018Distributed Distributional Deterministic Policy Gradientscited by 199
  2. 2020Acme: A Research Framework for Distributed Reinforcement Learningcited by 146
  3. 2018Reinforcement and Imitation Learning for Diverse Visuomotor Skillscited by 126
  4. 2019End-to-End Robotic Reinforcement Learning without Reward Engineeringcited by 122
  5. 2020Unifying Graph Convolutional Neural Networks and Label Propagationcited by 98
  6. 2019BAIL: Best-Action Imitation Learning for Batch Deep Reinforcement Learningcited by 91
  7. 2017Learning human behaviors from motion capture by adversarial imitationcited by 77
  8. 2020Offline Learning from Demonstrations and Unlabeled Experiencecited by 72
  9. 2016Unsupervised Perceptual Rewards for Imitation Learningcited by 65
  10. 2015Transductive Multi-view Zero-Shot Learningcited by 53
  11. 2019Task-Relevant Adversarial Imitation Learningcited by 45
  12. 2020Reward Propagation Using Graph Convolutional Networkscited by 15
  13. 2016Perceptual Reward Functionscited by 3

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Reward Evolution with Graph-of-Thoughts: A Bi-Level Language Model Framework for Reinforcement Learning
  2. 2023Reward-Directed Conditional Diffusion: Provable Distribution Estimation and Reward Improvement
  3. 2024Offline Reinforcement Learning with Imputed Rewards
  4. 2024Explaining Learned Reward Functions with Counterfactual Trajectories
  5. 2026From Connectivity to Rewards: Dense Reward Learning with Directed State Graphs
  6. 2025From Novelty to Imitation: Self-Distilled Rewards for Offline Reinforcement Learning
  7. 2026Causal Reward World Models: Zero-shot Reward Design for Automated Skill Generation
  8. 2025Shaping Sparse Rewards in Reinforcement Learning: A Semi-supervised Approach