mlsanityData to 7 Oct 2026

Non-Markovian Reward Modelling from Trajectory Labels via Interpretable Multiple Instance Learning

arXiv 2205.15367 ↗First posted 30 May 2022cs.LG cs.AIcited by 13

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

We generalise the problem of reward modelling (RM) for reinforcement learning (RL) to handle non-Markovian rewards. Existing work assumes that human evaluators observe each step in a trajectory independently when providing feedback on agent behaviour. In this work, we remove this assumption, extending RM to capture temporal dependencies in human assessment of trajectories. We show how RM can be approached as a multiple instance learning (MIL) problem, where trajectories are treated as bags with return labels, and steps within the trajectories are instances with unseen reward labels. We go on to develop new MIL models that are able to capture the time dependencies in labelled trajectories. We demonstrate on a range of RL tasks that our novel MIL models can reconstruct reward functions to a high level of accuracy, and can be used to train high-performing agent policies.

Key facts. Cited by 13 papers in the map (data through 7 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 20 of its 52 references that are in the map. First posted on arXiv 30 May 2022.

Builds on

20 of 52 references.

Closest ancestors · 4

Same direction, newest first

  1. 2021Interpretable Preference-based Reinforcement Learning with Tree-Structured Reward Functions20 Dec
  2. 2019Learning Human Objectives by Evaluating Hypothetical Behavior5 Dec
  3. 2018Scalable agent alignment via reward modeling: a research direction19 Nov
  4. 2017Deep reinforcement learning from human preferences12 Jun

Other references · 16

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2016Densely Connected Convolutional Networkscited by 6,691
  3. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  4. 2015Deep Reinforcement Learning with Double Q-learningcited by 1,704
  5. 2018Attention-based Deep Multiple Instance Learningcited by 1,077
  6. 2021TransMIL: Transformer based Correlated Multiple Instance Learning for Whole Slide Image Classificationcited by 549
  7. 2015Deep Recurrent Q-Learning for Partially Observable MDPscited by 457
  8. 2017Inverse Reward Designcited by 246
  9. 2017Deep Learning is Robust to Massive Label Noisecited by 164
  10. 2016Multiple Instance Learning: A Survey of Problem Characteristics and Applicationscited by 163
  11. 2016Revisiting Multiple Instance Neural Networkscited by 119
  12. 2020Reward-rational (implicit) choice: A unifying formalism for reward learningcited by 119
4 more
  1. 2021B-Pref: Benchmarking Preference-Based Reinforcement Learningcited by 106
  2. 2019Multiple instance learning with graph neural networkscited by 25
  3. 2022Model Agnostic Interpretability for Multiple Instance Learningcited by 10
  4. 2020Interactive Robot Training for Non-Markov Taskscited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 6; 2024: 4; 2025: 2; 2026: 1.

13 papers.

Sort:

By area:

  1. 2023Preference Transformer: Modeling Human Preferences using Transformers for RLcited by 77
  2. 2023Inverse Preference Learning: Preference-based RL without a Reward Functioncited by 68
  3. 2023Direct Preference-based Policy Optimization without Reward Modelingcited by 36
  4. 2023Inherently Interpretable Time Series Classification via Multiple Instance Learningcited by 23
  5. 2024Efficient Preference-based Reinforcement Learning via Aligned Experience Estimationcited by 9
  6. 2025A Statistical Case Against Empirical Human-AI Alignmentcited by 6
  7. 2024Detecting Hidden Triggers: Mapping Non-Markov Reward Functions to Markovcited by 5
  8. 2023Limited Resource Allocation in a Non-Markovian World: The Case of Maternal and Child Healthcarecited by 4
  9. 2025Policy Learning from Large Vision-Language Model Feedback without Reward Modelingcited by 4
  10. 2023TranDRL: A Transformer-Driven Deep Reinforcement Learning Enabled Prescriptive Maintenance Frameworkcited by 2
  11. 2024Beyond Simple Sum of Delayed Rewards: Non-Markovian Reward Modeling for Reinforcement Learningcited by 2
  12. 2024Reinforcement Learning from Bagged Rewardcited by 0
1 more
  1. 2026SP3O: Reinforcement Learning from Segment Preferences without Reward Modelingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024A Theoretical Framework for Partially Observed Reward-States in RLHF
  2. 2022Symbol Guided Hindsight Priors for Reward Learning from Human Preferences
  3. 2020Learning Non-Markovian Reward Models in MDPs
  4. 2023Exploiting Unlabeled Data for Feedback Efficient Human Preference based Reinforcement Learning
  5. 2019Reinforcement Learning with Non-Markovian Rewards
  6. 2026Reinforcement Learning from Human Feedback: A Statistical Perspective
  7. 2025RLAF: Reinforcement Learning from Automaton Feedback
  8. 2024Explaining Learned Reward Functions with Counterfactual Trajectories