mlsanityData to 14 Oct 2026

ACT: Empowering Decision Transformer with Dynamic Programming via Advantage Conditioning

arXiv 2309.05915 ↗First posted 12 Sep 2023cs.LG cs.AIcited by 25

On the map: Offline RL and imitation learning from data → Decision transformers and sequence modeling

Abstract (arXiv)

Decision Transformer (DT), which employs expressive sequence modeling techniques to perform action generation, has emerged as a promising approach to offline policy optimization. However, DT generates actions conditioned on a desired future return, which is known to bear some weaknesses such as the susceptibility to environmental stochasticity. To overcome DT's weaknesses, we propose to empower DT with dynamic programming. Our method comprises three steps. First, we employ in-sample value iteration to obtain approximated value functions, which involves dynamic programming over the MDP structure. Second, we evaluate action quality in context with estimated advantages. We introduce two types of advantage estimators, IAE and GAE, which are suitable for different tasks. Third, we train an Advantage-Conditioned Transformer (ACT) to generate actions conditioned on the estimated advantages. Finally, during testing, ACT generates actions conditioned on a desired advantage. Our evaluation results validate that, by leveraging the power of dynamic programming, ACT demonstrates effective trajectory stitching and robust action generation in spite of the environmental stochasticity, outperforming baseline methods across various benchmarks. Additionally, we conduct an in-depth analysis of ACT's various design choices through ablation studies. Our code is available at https://github.com/LAMDA-RL/ACT.

Key facts. Cited by 25 papers in the map (data through 14 Oct 2026). Placed in Decision transformers and sequence modeling, one of 13 directions in Offline RL and imitation learning from data. Builds on 38 of its 49 references that are in the map. First posted on arXiv 12 Sep 2023.

Builds on

38 of 49 references.

Closest ancestors · 6

Same direction, newest first

  1. 2023Masked Trajectory Models for Prediction, Representation, and Control4 May
  2. 2022Dichotomy of Control: Separating What You Can Control from What You Cannot24 Oct
  3. 2022Q-learning Decision Transformer: Leveraging Dynamic Programming for Conditional Sequence Modelling in Offline RL8 Sep
  4. 2022You Can't Count on Luck: Why Decision Transformers and RvS Fail in Stochastic Environments31 May
  5. 2022Online Decision Transformer11 Feb
  6. 2021Decision Transformer: Reinforcement Learning via Sequence Modeling2 Jun

Other references · 32

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2017Proximal Policy Optimization Algorithmscited by 15,215
  4. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  5. 2015Trust Region Policy Optimizationcited by 2,580
  6. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,417
  7. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 2,026
  8. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,920
  9. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,809
  10. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,427
  11. 2021Offline Reinforcement Learning with Implicit Q-Learningcited by 1,293
  12. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,241
20 more
  1. 2021A Minimalist Approach to Offline Reinforcement Learningcited by 899
  2. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 840
  3. 2020MOPO: Model-based Offline Policy Optimizationcited by 636
  4. 2021COMBO: Conservative Offline Model-Based Policy Optimizationcited by 370
  5. 2021Uncertainty-Based Offline Reinforcement Learning with Diversified Q-Ensemblecited by 302
  6. 2021RvS: What is Essential for Offline RL via Supervised Learning?cited by 185
  7. 2021Offline RL Without Off-Policy Evaluationcited by 151
  8. 2019Reinforcement Learning Upside Down: Don't Predict Rewards -- Just Map Them to Actionscited by 125
  9. 2022CORL: Research-oriented Deep Offline Reinforcement Learning Librarycited by 119
  10. 2023Extreme Q-Learning: MaxEnt RL without Entropycited by 116
  11. 2019Reward-Conditioned Policiescited by 102
  12. 2022When does return-conditioned supervised learning work for offline reinforcement learning?cited by 76
  13. 2021A Workflow for Offline Model-Free Robotic Reinforcement Learningcited by 69
  14. 2023Elastic Decision Transformercited by 67
  15. 2023SMART: Self-supervised Multi-task pretrAining with contRol Transformerscited by 44
  16. 2023Efficient Deep Reinforcement Learning Requires Regulating Overfittingcited by 36
  17. 2022UniMASK: Unified Inference in Sequential Decision Problemscited by 24
  18. 2022Addressing Optimism Bias in Sequence Modeling for Reinforcement Learningcited by 22
  19. 2023Large Sequence Models for Sequential Decision-Making: A Surveycited by 19
  20. 2023Distance Weighted Supervised Learning for Offline Interaction Datacited by 18

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 2; 2024: 7; 2025: 9; 2026: 7.

25 papers.

Sort:

By area:

  1. 2025Interactive Post-Training for Vision-Language-Action Modelscited by 135
  2. 2025Generative Trajectory Stitching through Diffusion Compositioncited by 41
  3. 2025Behavior-Regularized Diffusion Policy Optimization for Offline Reinforcement Learningcited by 20
  4. 2024Reinforced In-Context Black-Box Optimizationcited by 14
  5. 2024A2PO: Towards Effective Offline Reinforcement Learning from an Advantage-aware Perspectivecited by 11
  6. 2023Generalizable Task Representation Learning for Offline Meta-Reinforcement Learning with Data Limitationscited by 10
  7. 2023Uncertainty-Aware Decision Transformer for Stochastic Driving Environmentscited by 8
  8. 2024Adaptive Q-Aid for Conditional Supervised Learning in Offline Reinforcement Learningcited by 7
  9. 2024BeTAIL: Behavior Transformer Adversarial Imitation Learning from Human Racing Gameplaycited by 5
  10. 2024Goal-Conditioned Data Augmentation for Offline Reinforcement Learningcited by 4
  11. 2024Return-Aligned Decision Transformercited by 3
  12. 2024Debiased Offline Representation Learning for Fast Online Adaptation in Non-stationary Dynamicscited by 2
13 more
  1. 2025How to Provably Improve Return Conditioned Supervised Learning?cited by 1
  2. 2026Decoupling Return-to-Go for Efficient Decision Transformercited by 1
  3. 2025Model-Based Offline Reinforcement Learning with Reliability-Guaranteed Sequence Modelingcited by 0
  4. 2025Target Return Optimizer for Multi-Game Decision Transformercited by 0
  5. 2025Mildly Conservative Regularized Evaluation for Offline Reinforcement Learningcited by 0
  6. 2025Hybrid Sequence Modeling and Reinforced Verification for Controllable Target-Conditioned Decision Makingcited by 0
  7. 2025Target-Aligned Fusion for Decision-Sequence Learning under Dynamics Shiftcited by 0
  8. 2026Conditional Sequence Modeling for Safe Reinforcement Learningcited by 0
  9. 2026Advantage-based Temporal Attack in Reinforcement Learningcited by 0
  10. 2026Selecting Offline Reinforcement Learning Algorithms for Stochastic Network Controlcited by 0
  11. 2026Beyond Autoregressive RTG: Conditioning via Injection Outside Sequential Modeling in Decision Transformercited by 0
  12. 2026Trust Guided Decision Transformercited by 0
  13. 2026Reinforcement Learning via Advantage Guidancecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Q-value Regularized Transformer for Offline Reinforcement Learning
  2. 2025Adjusting the Output of Decision Transformer with Action Gradient
  3. 2023Rethinking Decision Transformer via Hierarchical Reinforcement Learning
  4. 2024Enhancing Decision Transformer with Diffusion-Based Trajectory Branch Generation
  5. 2024DODT: Enhanced Online Decision Transformer Learning through Dreamer's Actor-Critic Trajectory Forecasting
  6. 2024Understanding the Training and Generalization of Pretrained Transformer for Sequential Decision Making
  7. 2023Critic-Guided Decision Transformer for Offline Reinforcement Learning
  8. 2023Multi-Objective Decision Transformers for Offline Reinforcement Learning