mlsanityData to 7 Oct 2026

Reinformer: Max-Return Sequence Modeling for Offline RL

arXiv 2405.08740 ↗First posted 14 May 2024cs.LGcited by 25

On the map: Offline RL and imitation learning from data → Decision transformers and sequence modeling

Abstract (arXiv)

As a data-driven paradigm, offline reinforcement learning (RL) has been formulated as sequence modeling that conditions on the hindsight information including returns, goal or future trajectory. Although promising, this supervised paradigm overlooks the core objective of RL that maximizes the return. This overlook directly leads to the lack of trajectory stitching capability that affects the sequence model learning from sub-optimal data. In this work, we introduce the concept of max-return sequence modeling which integrates the goal of maximizing returns into existing sequence models. We propose Reinforced Transformer (Reinformer), indicating the sequence model is reinforced by the RL objective. Reinformer additionally incorporates the objective of maximizing returns in the training phase, aiming to predict the maximum future return within the distribution. During inference, this in-distribution maximum return will guide the selection of optimal actions. Empirically, Reinformer is competitive with classical RL methods on the D4RL benchmark and outperforms state-of-the-art sequence model particularly in trajectory stitching ability. Code is public at https://github.com/Dragon-Zhuang/Reinformer.

Key facts. Cited by 25 papers in the map (data through 7 Oct 2026). Placed in Decision transformers and sequence modeling, one of 13 directions in Offline RL and imitation learning from data. Builds on 33 of its 45 references that are in the map. First posted on arXiv 14 May 2024.

Builds on

33 of 45 references.

Closest ancestors · 6

Same direction, newest first

  1. 2023Decision ConvFormer: Local Filtering in MetaFormer is Sufficient for Decision Making4 Oct
  2. 2023Decision S4: Efficient Sequence-Based RL via State Spaces Layers8 Jun
  3. 2023Future-conditioned Unsupervised Pretraining for Decision Transformer26 May
  4. 2022Q-learning Decision Transformer: Leveraging Dynamic Programming for Conditional Sequence Modelling in Offline RL8 Sep
  5. 2022Multi-Game Decision Transformers30 May
  6. 2021Decision Transformer: Reinforcement Learning via Sequence Modeling2 Jun

Other references · 27

  1. 2017Attention Is All You Needcited by 59,932
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  4. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,873
  5. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,777
  6. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,406
  7. 2021Offline Reinforcement Learning with Implicit Q-Learningcited by 1,245
  8. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
  9. 2021A Minimalist Approach to Offline Reinforcement Learningcited by 882
  10. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 833
  11. 2021Deep Reinforcement Learning at the Edge of the Statistical Precipicecited by 781
  12. 2019Large Batch Optimization for Deep Learning: Training BERT in 76 minutescited by 671
15 more
  1. 2019Behavior Regularized Offline Reinforcement Learningcited by 654
  2. 2020Critic Regularized Regressioncited by 269
  3. 2021Offline Reinforcement Learning with Fisher Divergence Critic Regularizationcited by 237
  4. 2021Offline RL Without Off-Policy Evaluationcited by 150
  5. 2022Pessimistic Bootstrapping for Uncertainty-Driven Offline Reinforcement Learningcited by 125
  6. 2019Reinforcement Learning Upside Down: Don't Predict Rewards -- Just Map Them to Actionscited by 124
  7. 2022CORL: Research-oriented Deep Offline Reinforcement Learning Librarycited by 117
  8. 2019Training Agents using Upside-Down Reinforcement Learningcited by 106
  9. 2022When does return-conditioned supervised learning work for offline reinforcement learning?cited by 73
  10. 2022Offline Q-Learning on Diverse Multi-Task Data Both Scales And Generalizescited by 72
  11. 2023Elastic Decision Transformercited by 66
  12. 2023Behavior Proximal Policy Optimizationcited by 43
  13. 2021Regularized Behavior Value Estimationcited by 32
  14. 2023Beyond OOD State Actions: Supported Cross-Domain Offline Reinforcement Learningcited by 27
  15. 2023Design from Policies: Conservative Test-Time Adaptation for Offline Policy Optimizationcited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 7; 2025: 12; 2026: 6.

25 papers.

Sort:

By area:

  1. 2025ReinboT: Amplifying Robot Visual-Language Manipulation with Reinforcement Learningcited by 44
  2. 2025Generative Trajectory Stitching through Diffusion Compositioncited by 38
  3. 2024Return Augmented Decision Transformer for Off-Dynamics Reinforcement Learningcited by 13
  4. 2025Vintix: Action Model via In-Context Reinforcement Learningcited by 8
  5. 2025Yes, Q-learning Helps Offline In-Context RLcited by 8
  6. 2024Tackling Data Corruption in Offline Reinforcement Learning via Sequence Modelingcited by 7
  7. 2024Pre-trained Language Models Improve the Few-shot Prompt Ability of Decision Transformercited by 6
  8. 2025TDMPBC: Self-Imitative Reinforcement Learning for Humanoid Robot Controlcited by 6
  9. 2026Flow Actor-Critic for Offline Reinforcement Learningcited by 6
  10. 2024DIDI: Diffusion-Guided Diversity for Offline Behavioral Generationcited by 4
  11. 2024Q-value Regularized Decision ConvFormer for Offline Reinforcement Learningcited by 3
  12. 2024Goal-Conditioned Supervised Learning for Multi-Objective Recommendationcited by 1
13 more
  1. 2025PIQL: Projective Implicit Q-Learning with Support Constraint for Offline Reinforcement Learningcited by 1
  2. 2025Beyond the Known: Decision Making with Counterfactual Reasoning Decision Transformercited by 1
  3. 2025How to Provably Improve Return Conditioned Supervised Learning?cited by 1
  4. 2025Offline Reinforcement Learning with Wasserstein Regularization via Optimal Transport Mapscited by 1
  5. 2025Boundary-to-Region Supervision for Offline Safe Reinforcement Learningcited by 1
  6. 2025MUVLA: Learning to Explore Object Navigation via Map Understandingcited by 1
  7. 2026Online Finetuning Decision Transformers with Pure RL Gradientscited by 1
  8. 2026Compositional Transduction with Latent Analogies for Offline Goal-Conditioned Reinforcement Learningcited by 1
  9. 2024Decision MetaMamba: Enhancing Selective SSM in Offline RL with Heterogeneous Sequence Mixingcited by 0
  10. 2025Adjusting the Output of Decision Transformer with Action Gradientcited by 0
  11. 2026GAS: Enhancing Reward-Cost Balance of Generative Model-assisted Offline Safe RLcited by 0
  12. 2026Decision MetaMamba: Enhancing Selective SSM in Offline RL with Heterogeneous Sequence Mixingcited by 0
  13. 2026IPD: Boosting Sequential Policy with Imaginary Planning Distillation in Offline Reinforcement Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2021Offline Reinforcement Learning as One Big Sequence Modeling Problem
  2. 2022Bootstrapped Transformer for Offline Reinforcement Learning
  3. 2023Multi-Objective Decision Transformers for Offline Reinforcement Learning
  4. 2024Context-Former: Stitching via Latent Conditioned Sequence Modeling
  5. 2023SaFormer: A Conditional Sequence Modeling Approach to Offline Safe Reinforcement Learning
  6. 2024In-Dataset Trajectory Return Regularization for Offline Preference-based Reinforcement Learning
  7. 2023A Tractable Inference Perspective of Offline RL
  8. 2024Q-value Regularized Transformer for Offline Reinforcement Learning