mlsanityData to 14 Oct 2026

Beyond the Known: Decision Making with Counterfactual Reasoning Decision Transformer

arXiv 2505.09114 ↗First posted 14 May 2025cs.AI cs.LGcited by 1

On the map: Offline RL and imitation learning from data → Decision transformers and sequence modeling

Abstract (arXiv)

Decision Transformers (DT) play a crucial role in modern reinforcement learning, leveraging offline datasets to achieve impressive results across various domains. However, DT requires high-quality, comprehensive data to perform optimally. In real-world applications, the lack of training data and the scarcity of optimal behaviours make training on offline datasets challenging, as suboptimal data can hinder performance. To address this, we propose the Counterfactual Reasoning Decision Transformer (CRDT), a novel framework inspired by counterfactual reasoning. CRDT enhances DT ability to reason beyond known data by generating and utilizing counterfactual experiences, enabling improved decision-making in unseen scenarios. Experiments across Atari and D4RL benchmarks, including scenarios with limited data and altered dynamics, demonstrate that CRDT outperforms conventional DT approaches. Additionally, reasoning counterfactually allows the DT agent to obtain stitching abilities, combining suboptimal trajectories, without architectural modifications. These results highlight the potential of counterfactual reasoning to enhance reinforcement learning agents' performance and generalization capabilities.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Decision transformers and sequence modeling, one of 13 directions in Offline RL and imitation learning from data. Builds on 32 of its 48 references that are in the map. First posted on arXiv 14 May 2025.

Builds on

32 of 48 references.

Closest ancestors · 7

Same direction, newest first

  1. 2024Reinformer: Max-Return Sequence Modeling for Offline RL14 May
  2. 2023Decision ConvFormer: Local Filtering in MetaFormer is Sufficient for Decision Making4 Oct
  3. 2023Future-conditioned Unsupervised Pretraining for Decision Transformer26 May
  4. 2023When should we prefer Decision Transformers for Offline Reinforcement Learning?23 May
  5. 2022Q-learning Decision Transformer: Leveraging Dynamic Programming for Conditional Sequence Modelling in Offline RL8 Sep
  6. 2022Online Decision Transformer11 Feb
  7. 2021Decision Transformer: Reinforcement Learning via Sequence Modeling2 Jun

Other references · 25

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,920
  3. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,809
  4. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,427
  5. 2021Offline Reinforcement Learning with Implicit Q-Learningcited by 1,293
  6. 2020MOPO: Model-based Offline Policy Optimizationcited by 636
  7. 2020MOReL : Model-Based Offline Reinforcement Learningcited by 590
  8. 2021Bellman-consistent Pessimism for Offline Reinforcement Learningcited by 269
  9. 2021Offline Reinforcement Learning with Fisher Divergence Critic Regularizationcited by 238
  10. 2019Reinforcement Learning Upside Down: Don't Predict Rewards -- Just Map Them to Actionscited by 125
  11. 2022Causal Transformer for Estimating Counterfactual Outcomescited by 123
  12. 2019Training Agents using Upside-Down Reinforcement Learningcited by 107
13 more
  1. 2018Supervised Reinforcement Learning with Recurrent Neural Network for Dynamic Treatment Recommendationcited by 101
  2. 2020Counterfactual Data Augmentation using Locally Factored Dynamicscited by 93
  3. 2021Causal Influence Detection for Improving Efficiency in Reinforcement Learningcited by 71
  4. 2023Elastic Decision Transformercited by 67
  5. 2020Counterfactual Credit Assignment in Model-Free Reinforcement Learningcited by 64
  6. 2022MoCoDA: Model-based Counterfactual Data Augmentationcited by 44
  7. 2022Estimating average causal effects from patient trajectoriescited by 29
  8. 2020Counterfactually Guided Off-policy Transfer in Clinical Settingscited by 14
  9. 2020G-Net: A Deep Learning Approach to G-computation for Counterfactual Outcome Prediction Under Dynamic Treatment Regimescited by 13
  10. 2021End-to-End Balancing for Causal Continuous Treatment-Effect Estimationcited by 9
  11. 2024Policy Learning for Off-Dynamics RL with Deficient Supportcited by 9
  12. 2023Offline Imitation Learning with Variational Counterfactual Reasoningcited by 7
  13. 2024Offline Trajectory Optimization for Offline Reinforcement Learningcited by 1

Built on by

Cited by year

Citing papers per year: 2025: 1.

1 paper.

  1. 2025DmC: Nearest Neighbor Guidance Diffusion Model for Offline Cross-domain Reinforcement Learningcited by 7

Similar, not cited

Close in wording, no citation either way

  1. 2021Transfer learning with causal counterfactual reasoning in Decision Transformers
  2. 2024Solving Continual Offline Reinforcement Learning with Decision Transformer
  3. 2025Counterfactual Explanations for Continuous Action Reinforcement Learning
  4. 2024Explaining Reinforcement Learning: A Counterfactual Shapley Values Approach
  5. 2025A Comparison Between Decision Transformers and Traditional Offline Reinforcement Learning Algorithms
  6. 2025Should We Ever Prefer Decision Transformer for Offline Reinforcement Learning?
  7. 2024Out-of-Distribution Adaptation in Offline RL: Counterfactual Reasoning via Causal Normalizing Flows
  8. 2025An Identifiable Cost-Aware Causal Decision-Making Framework Using Counterfactual Reasoning