mlsanityData to 14 Oct 2026

Themis: An explainable AI-enabled framework for Reinforcement Learning with Human Feedback

arXiv 2606.24622 ↗First posted 23 Jun 2026cs.AI cs.HCcited by 0

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

also in 1 more
Abstract (arXiv)

Training safe Reinforcement Learning (RL) systems is inherently challenging, with no guarantee of avoiding unwanted behaviors. The most effective defenses against this are (i) transparency through explainability and (ii) alignment via human feedback. While both show promising results, no publicly available framework currently combines them. To address this, we introduce Themis, an XAI-enabled testing and evaluation framework for Reinforcement Learning from Human Feedback. Themis supports over 200 widely used environments and is easily configurable for experiments in RL, transparency, and alignment. Our results show that Themis can train reward models that match or outperform the environment's true reward signal using human preferences. We also provide a cloud-based platform for collecting human feedback and managing experiments. It is user-friendly, auto-scalable, and supports large participant groups across multiple experiments without extra development overhead. Tests show Themis can support one thousand users in back-to-back experiments on a modest commercial machine.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 32 of its 50 references that are in the map. First posted on arXiv 23 Jun 2026.

Builds on

32 of 50 references.

Closest ancestors · 8

Same direction, newest first

  1. 2025Maximizing the efficiency of human feedback in AI alignment: a comparative analysis16 Nov
  2. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback27 Jul
  3. 2022SURF: Semi-supervised Reward Learning with Data Augmentation for Feedback-efficient Preference-based Reinforcement Learning18 Mar
  4. 2021PEBBLE: Feedback-Efficient Interactive Reinforcement Learning via Relabeling Experience and Unsupervised Pre-training9 Jun
  5. 2020Weak Human Preference Supervision For Deep Reinforcement Learning25 Jul
  6. 2018Reward learning from human preferences and demonstrations in Atari15 Nov
  7. 2018DQN-TAMER: Human-in-the-Loop Reinforcement Learning with Intractable Feedback28 Oct
  8. 2017Deep reinforcement learning from human preferences12 Jun

Other references · 24

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2017A Unified Approach to Interpreting Model Predictionscited by 6,555
  3. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  4. 2017Axiomatic Attribution for Deep Networkscited by 3,384
  5. 2015Trust Region Policy Optimizationcited by 2,580
  6. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,258
  7. 2020Deep Reinforcement Learning for Autonomous Driving: A Surveycited by 721
  8. 2020Estimating Training Data Influence by Tracing Gradient Descentcited by 580
  9. 2020Captum: A unified and generic model interpretability library for PyTorchcited by 547
  10. 2023Minigrid & Miniworld: Modular & Customizable Reinforcement Learning Environments for Goal-Oriented Taskscited by 327
  11. 2018BabyAI: A Platform to Study the Sample Efficiency of Grounded Language Learningcited by 244
  12. 2022Defining and Characterizing Reward Hackingcited by 203
12 more
  1. 2024Deep Reinforcement Learning for Robotics: A Survey of Real-World Successescited by 176
  2. 2019Soft Actor-Critic for Discrete Action Settingscited by 174
  3. 2017Visualizing and Understanding Atari Agentscited by 141
  4. 2021B-Pref: Benchmarking Preference-Based Reinforcement Learningcited by 106
  5. 2019Leveraging Human Guidance for Deep Reinforcement Learning Taskscited by 42
  6. 2020Widening the Pipeline in Human-Guided Reinforcement Learning with Explanation and Context-Aware Data Augmentationcited by 42
  7. 2021A Survey on Interactive Reinforcement Learning: Design Principles and Open Challengescited by 41
  8. 2023Explainable Deep Reinforcement Learning: State of the Art and Challengescited by 33
  9. 2020Local and Global Explanations of Agent Behavior: Integrating Strategy Summaries with Saliency Mapscited by 30
  10. 2022The Utility of Explainable AI in Ad Hoc Human-Machine Teamingcited by 30
  11. 2021Collective eXplainable AI: Explaining Cooperative Strategies and Agent Contribution in Multiagent Reinforcement Learning with Shapley Valuescited by 25
  12. 2021Human-in-the-Loop Deep Reinforcement Learning with Application to Autonomous Drivingcited by 16

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Mapping out the Space of Human Feedback for Reinforcement Learning: A Conceptual Framework
  2. 2024AI Alignment through Reinforcement Learning from Human Feedback? Contradictions and Limitations
  3. 2026Reinforcement Learning from Human Feedback: A Statistical Perspective
  4. 2025SHARPIE: A Modular Framework for Reinforcement Learning and Human-AI Interaction Experiments
  5. 2024Uni-RLHF: Universal Platform and Benchmark Suite for Reinforcement Learning with Diverse Human Feedback
  6. 2025Reinforcement Learning from Human Feedback
  7. 2023A Survey of Reinforcement Learning from Human Feedback
  8. 2025RLBFF: Binary Flexible Feedback to bridge between Human Feedback & Verifiable Rewards