TECM*: A Data-Driven Assessment to Reinforcement Learning Methods and Application to Heparin Treatment Strategy for Surgical Sepsis
Not on the map. Why?
Abstract (arXiv)
Objective: Sepsis is a life-threatening condition caused by severe infection leading to acute organ dysfunction. This study proposes a data-driven metric and a continuous reward function to optimize personalized heparin therapy in surgical sepsis patients. Methods: Data from the MIMIC-IV v1.0 and eICU v2.0 databases were used for model development and evaluation. The training cohort consisted of abdominal surgery patients receiving unfractionated heparin (UFH) after postoperative sepsis onset. We introduce a new RL-based framework: converting the discrete SOFA score to a continuous cxSOFA for more nuanced state and reward functions; Second, defining "good" or "bad" strategies based on cxSOFA by a stepwise manner; Third, proposing a Treatment Effect Comparison Matrix (TECM), analogous to a confusion matrix for classification tasks, to evaluate the treatment strategies. We applied different RL algorithms, Q-Learning, DQN, DDQN, BCQ and CQL to optimize the treatment and comprehensively evaluated the framework. Results: Among the AI-derived strategies, the cxSOFA-CQL model achieved the best performance, reducing mortality from 1.83% to 0.74% with the average hospital stay from 11.11 to 9.42 days. TECM demonstrated consistent outcomes across models, highlighting robustness. Conclusion: The proposed RL framework enables interpretable and robust optimization of heparin therapy in surgical sepsis. Continuous cxSOFA scoring and TECM-based evaluation provide nuanced treatment assessment, showing promise for improving clinical outcomes and decision-support reliability.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 11 of its 61 references that are in the map. First posted on arXiv 2 Dec 2025.
Builds on
11 of 61 references.
References · 11
- 2013Playing Atari with Deep Reinforcement Learningcited by 3,274
- 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,809
- 2015Deep Reinforcement Learning with Double Q-learningcited by 1,745
- 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,241
- 2015Prioritized Experience Replaycited by 1,099
- 2019Reinforcement Learning in Healthcare: A Surveycited by 310
- 2020Secure and Robust Machine Learning for Healthcare: A Surveycited by 80
- 2017Deep Reinforcement Learning for Sepsis Treatmentcited by 71
- 2018Deep Reinforcement Learning for Dynamic Treatment Regimes on Medical Registry Datacited by 23
- 2020Optimizing Medical Treatment for Sepsis in Intensive Care: from Reinforcement Learning to Pre-Trial Evaluationcited by 8
- 2019Performance Dynamics and Termination Errors in Reinforcement Learning: A Unifying Perspectivecited by 2
Built on by
Citing papers per year: 2026: 1.
1 paper.
Similar, not cited
Close in wording, no citation either way
- 2021Offline reinforcement learning with uncertainty for treatment strategies in sepsis
- 2026Learning Preference-Based Objectives from Clinical Narratives for Dynamic Sepsis Treatment
- 2024Empowering Clinicians with Medical Decision Transformers: A Framework for Sepsis Treatment
- 2024Development and Validation of Heparin Dosing Policies Using an Offline Reinforcement Learning Algorithm
- 2024ICU-Sepsis: A Benchmark MDP Built from Real Medical Data
- 2025Exploring a Graph-based Approach to Offline Reinforcement Learning for Sepsis Treatment
- 2024How Consistent are Clinicians? Evaluating the Predictability of Sepsis Disease Progression with Dynamics Models
- 2024Reinforced Sequential Decision-Making for Sepsis Treatment: The POSNEGDM Framework with Mortality Classifier and Transformer