mlsanityData to 14 Oct 2026

Off-Policy Selection for Initiating Human-Centric Experimental Design

arXiv 2410.20017 ↗First posted 26 Oct 2024cs.LG cs.AI cs.HCcited by 0

Not on the map. Why?

Abstract (arXiv)

In human-centric tasks such as healthcare and education, the heterogeneity among patients and students necessitates personalized treatments and instructional interventions. While reinforcement learning (RL) has been utilized in those tasks, off-policy selection (OPS) is pivotal to close the loop by offline evaluating and selecting policies without online interactions, yet current OPS methods often overlook the heterogeneity among participants. Our work is centered on resolving a pivotal challenge in human-centric systems (HCSs): how to select a policy to deploy when a new participant joining the cohort, without having access to any prior offline data collected over the participant? We introduce First-Glance Off-Policy Selection (FPS), a novel approach that systematically addresses participant heterogeneity through sub-group segmentation and tailored OPS criteria to each sub-group. By grouping individuals with similar traits, FPS facilitates personalized policy selection aligned with unique characteristics of each participant or group of participants. FPS is evaluated via two important but challenging applications, intelligent tutoring systems and a healthcare application for sepsis treatment and intervention. FPS presents significant advancement in enhancing learning outcomes of students and in-hospital care outcomes.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 48 of its 83 references that are in the map. First posted on arXiv 26 Oct 2024.

Builds on

48 of 83 references.

References · 48

  1. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  2. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,809
  3. 2019Dream to Control: Learning Behaviors by Latent Imaginationcited by 1,653
  4. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,546
  5. 2015Doubly Robust Off-policy Value Evaluation for Reinforcement Learningcited by 339
  6. 2016Data-Efficient Off-Policy Policy Evaluation for Reinforcement Learningcited by 286
  7. 2019DualDICE: Behavior-Agnostic Estimation of Discounted Stationary Distribution Correctionscited by 254
  8. 2019Batch Policy Learning under Constraintscited by 244
  9. 2019Stochastic Latent Actor-Critic: Deep Reinforcement Learning with a Latent Variable Modelcited by 231
  10. 2018Breaking the Curse of Horizon: Infinite-Horizon Off-Policy Estimationcited by 227
  11. 2019Minimax Weight and Q-Function Learning for Off-Policy Evaluationcited by 150
  12. 2019Empirical Study of Off-Policy Policy Evaluation for Reinforcement Learningcited by 121
36 more
  1. 2019Counterfactual Off-Policy Evaluation with Gumbel-Max Structural Causal Modelscited by 117
  2. 2020Hyperparameter Selection for Offline Reinforcement Learningcited by 114
  3. 2020Batch Value-function Approximation with Only Realizabilitycited by 104
  4. 2021Benchmarks for Deep Off-Policy Evaluationcited by 89
  5. 2020Off-Policy Evaluation via the Regularized Lagrangiancited by 86
  6. 2020Model Patching: Closing the Subgroup Performance Gap with Data Augmentationcited by 83
  7. 2017Deep Reinforcement Learning for Sepsis Treatmentcited by 71
  8. 2020GradientDICE: Rethinking Generalized Offline Estimation of Stationary Valuescited by 70
  9. 2021A Workflow for Offline Model-Free Robotic Reinforcement Learningcited by 69
  10. 2021Model Selection for Offline Reinforcement Learning: Practical Considerations for Healthcare Settingscited by 68
  11. 2020Off-policy Policy Evaluation For Sequential Decisions Under Unobserved Confoundingcited by 61
  12. 2017Toeplitz Inverse Covariance-Based Clustering of Multivariate Time Series Datacited by 47
  13. 2018Policy Optimization via Importance Samplingcited by 47
  14. 2021Bootstrapping Fitted Q-Evaluation for Off-Policy Inferencecited by 40
  15. 2020Adaptive Estimator Selection for Off-Policy Evaluationcited by 38
  16. 2021Autoregressive Dynamics Models for Offline Policy Evaluation and Optimizationcited by 33
  17. 2021Towards Hyperparameter-free Policy Selection for Offline Reinforcement Learningcited by 32
  18. 2020Offline Policy Selection under Uncertaintycited by 31
  19. 2021Model-Based Reinforcement Learning via Latent-Space Collocationcited by 28
  20. 2019Open-Sourced Reinforcement Learning Environments for Surgical Roboticscited by 26
  21. 2021Minimax Model Learningcited by 16
  22. 2021Learning Generalized Gumbel-max Causal Mechanismscited by 16
  23. 2021FaiR-IoT: Fairness-aware Human-in-the-Loop Reinforcement Learning for Harnessing Human Variability in Personalized IoTcited by 15
  24. 2021Robust On-Policy Sampling for Data-Efficient Policy Evaluation in Reinforcement Learningcited by 15
  25. 2022Data-Efficient Pipeline for Offline Reinforcement Learning with Limited Datacited by 12
  26. 2021Model Selection in Batch Policy Optimizationcited by 11
  27. 2023Reinforcement Learning Tutor Better Supported Lower Performers in a Math Taskcited by 11
  28. 2023Leveraging Deep Reinforcement Learning for Metacognitive Interventions across Intelligent Tutoring Systemscited by 10
  29. 2024Off-Policy Evaluation in Markov Decision Processes under Weak Distributional Overlapcited by 8
  30. 2021Early Performance Prediction using Interpretable Patterns in Programming Process Datacited by 7
  31. 2022Giving Feedback on Interactive Student Programs with Meta-Explorationcited by 6
  32. 2023Variational Latent Branching Model for Off-Policy Evaluationcited by 6
  33. 2023Off-Policy Evaluation for Human Feedbackcited by 6
  34. 2021Identification of Subgroups With Similar Benefits in Off-Policy Policy Evaluationcited by 5
  35. 2023HOPE: Human-Centric Off-Policy Evaluation for E-Learning and Healthcarecited by 5
  36. 2023Off-Policy Evaluation for Action-Dependent Non-Stationary Environmentscited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026JumpStart Your Policy Learning with Lessons from 160,000 Training Runs
  2. 2022Federated Offline Reinforcement Learning
  3. 2025Combining LLM decision and RL action selection to improve RL policy for adaptive interventions
  4. 2024OPERA: Automatic Offline Policy Evaluation with Re-weighted Aggregates of Multiple Estimators
  5. 2023Off-policy Evaluation in Doubly Inhomogeneous Environments
  6. 2024Reinforcement Learning in Dynamic Treatment Regimes Needs Critical Reexamination
  7. 2025Offline Policy Evaluation of Multi-Turn LLM Health Coaching with Real Users
  8. 2026Active Offline-to-Online Reinforcement Learning