mlsanityData to 7 Oct 2026

Enhancing RL Safety with Counterfactual LLM Reasoning

arXiv 2409.10188 ↗First posted 16 Sep 2024cs.LGcited by 6

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

Reinforcement learning (RL) policies may exhibit unsafe behavior and are hard to explain. We use counterfactual large language model reasoning to enhance RL policy safety post-training. We show that our approach improves and helps to explain the RL policy safety.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 13 of its 23 references that are in the map. First posted on arXiv 16 Sep 2024.

Builds on

13 of 23 references.

References · 13

  1. 2017Attention Is All You Needcited by 59,932
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2013Playing Atari with Deep Reinforcement Learningcited by 3,211
  4. 2023Guiding Pretraining in Reinforcement Learning with Large Language Modelscited by 185
  5. 2018Omega-Regular Objectives in Model-Free Reinforcement Learningcited by 74
  6. 2020The Probabilistic Model Checker Stormcited by 64
  7. 2021Scalar reward is not enough: A response to Silver, Singh, Precup and Sutton (2021)cited by 55
  8. 2022Safe Reinforcement Learning via Shielding under Partial Observabilitycited by 49
  9. 2019An Inductive Synthesis Framework for Verifiable Reinforcement Learningcited by 39
  10. 2022COOL-MC: A Comprehensive Tool for Reinforcement Learning and Model Checkingcited by 19
  11. 2023Explaining Reinforcement Learning Agents Through Counterfactual Action Outcomescited by 13
  12. 2021"I Don't Think So": Summarizing Policy Disagreements for Agent Comparisoncited by 6
1 more
  1. 2024Mental Modeling of Reinforcement Learning Agents by Language Modelscited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 2; 2026: 4.

6 papers.

  1. 2026Formally Verifying and Explaining Sepsis Treatment Policies with COOL-MCcited by 3
  2. 2025Co-Activation Graph Analysis of Safety-Verified and Explainable Deep Reinforcement Learning Policiescited by 2
  3. 2025Verifying Memoryless Sequential Decision-making of Large Language Modelscited by 1
  4. 2026COOL-MC: Verifying and Explaining RL Policies for Platelet Inventory Managementcited by 0
  5. 2026COOL-MC: Verifying and Explaining RL Policies for Multi-bridge Network Maintenancecited by 0
  6. 2026Automated Testing of LLM-Based Post Hoc Explainers Using Model Checking as an Oraclecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Breaking the Safety-Capability Tradeoff: Reinforcement Learning with Verifiable Rewards Maintains Safety Guardrails in LLMs
  2. 2025Beyond SFT: Reinforcement Learning for Safer Large Reasoning Models with Better Reasoning Ability
  3. 2025Part I: Tricks or Traps? A Deep Dive into RL for LLM Reasoning
  4. 2025RL-PLUS: Countering Capability Boundary Collapse of LLMs in Reinforcement Learning with Hybrid-policy Optimization
  5. 2026Rethinking RL for LLM Reasoning: It's Sparse Policy Selection, Not Capability Learning
  6. 2024Evaluating Defences against Unsafe Feedback in RLHF
  7. 2025Revisiting Reinforcement Learning for LLM Reasoning from A Cross-Domain Perspective
  8. 2025A Survey of Reinforcement Learning for Large Reasoning Models