mlsanityData to 14 Oct 2026

Time After Time: Deep-Q Effect Estimation for Interventions on When and What to do

arXiv 2503.15890 ↗First posted 20 Mar 2025cs.LG cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

Problems in fields such as healthcare, robotics, and finance requires reasoning about the value both of what decision or action to take and when to take it. The prevailing hope is that artificial intelligence will support such decisions by estimating the causal effect of policies such as how to treat patients or how to allocate resources over time. However, existing methods for estimating the effect of a policy struggle with irregular time. They either discretize time, or disregard the effect of timing policies. We present a new deep-Q algorithm that estimates the effect of both when and what to do called Earliest Disagreement Q-Evaluation (EDQ). EDQ makes use of recursion for the Q-function that is compatible with flexible sequence models, such as transformers. EDQ provides accurate estimates under standard assumptions. We validate the approach through experiments on survival time and tumor growth tasks.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 29 of its 59 references that are in the map. First posted on arXiv 20 Mar 2025.

Builds on

29 of 59 references.

References · 29

  1. 2015Deep Reinforcement Learning with Double Q-learningcited by 1,745
  2. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,546
  3. 2016Learning Representations for Counterfactual Inferencecited by 329
  4. 2016Safe and Efficient Off-Policy Reinforcement Learningcited by 264
  5. 2019Batch Policy Learning under Constraintscited by 244
  6. 2020Estimating Counterfactual Treatment Outcomes over Time Through Adversarially Balanced Representationscited by 155
  7. 2022Causal Transformer for Estimating Counterfactual Outcomescited by 123
  8. 2019Empirical Study of Off-Policy Policy Evaluation for Reinforcement Learningcited by 121
  9. 2019Counterfactual Off-Policy Evaluation with Gumbel-Max Structural Causal Modelscited by 117
  10. 2020Hyperparameter Selection for Offline Reinforcement Learningcited by 114
  11. 2022A Review of Off-Policy Evaluation in Reinforcement Learningcited by 95
  12. 2022q-Learning in Continuous Timecited by 91
17 more
  1. 2021Benchmarks for Deep Off-Policy Evaluationcited by 89
  2. 2019Off-Policy Evaluation in Partially Observable Environmentscited by 66
  3. 2020Off-policy Policy Evaluation For Sequential Decisions Under Unobserved Confoundingcited by 61
  4. 2022Continuous-Time Modeling of Counterfactual Outcomes Using Neural Controlled Differential Equationscited by 59
  5. 2019Learning When-to-Treat Policiescited by 49
  6. 2016Q(λ) with Off-Policy Correctionscited by 48
  7. 2023Event Stream GPT: A Data Pre-processing and Modeling Library for Generative, Pre-trained Transformers over Continuous-time Sequences of Complex Eventscited by 46
  8. 2017Multi-step Reinforcement Learning: A Unifying Algorithmcited by 42
  9. 2018Deep Reinforcement Learning of Marked Temporal Point Processescited by 33
  10. 2020Probabilistic Machine Learning for Healthcarecited by 23
  11. 2022Bellman Meets Hawkes: Model-Based Reinforcement Learning via Temporal Point Processescited by 23
  12. 2015Generalized Emphatic Temporal Difference Learning: Bias-Variance Analysiscited by 17
  13. 2023Accounting For Informative Sampling When Learning to Forecast Treatment Outcomes Over Timecited by 15
  14. 2021Continuous-time targeted minimum loss-based estimation of intervention-specific mean outcomescited by 9
  15. 2024G-Transformer: Counterfactual Outcome Prediction under Dynamic and Time-varying Treatment Regimescited by 8
  16. 2022From algorithms to action: improving patient care requires causalitycited by 5
  17. 2022Graphical criteria for the identification of marginal causal effects in continuous-time survival and event-history analysescited by 4

Built on by

Not cited yet.