mlsanityData to 7 Oct 2026

Addressing Optimism Bias in Sequence Modeling for Reinforcement Learning

arXiv 2207.10295 ↗First posted 21 Jul 2022cs.LG cs.AI cs.ROcited by 22

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

Impressive results in natural language processing (NLP) based on the Transformer neural network architecture have inspired researchers to explore viewing offline reinforcement learning (RL) as a generic sequence modeling problem. Recent works based on this paradigm have achieved state-of-the-art results in several of the mostly deterministic offline Atari and D4RL benchmarks. However, because these methods jointly model the states and actions as a single sequencing problem, they struggle to disentangle the effects of the policy and world dynamics on the return. Thus, in adversarial or stochastic environments, these methods lead to overly optimistic behavior that can be dangerous in safety-critical systems like autonomous driving. In this work, we propose a method that addresses this optimism bias by explicitly disentangling the policy and world models, which allows us at test time to search for policies that are robust to multiple possible futures in the environment. We demonstrate our method's superior performance on a variety of autonomous driving tasks in simulation.

Key facts. Cited by 22 papers in the map (data through 7 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 29 of its 38 references that are in the map. First posted on arXiv 21 Jul 2022.

Builds on

29 of 38 references.

Closest ancestors · 7

Same direction, newest first

  1. 2021Offline Reinforcement Learning with Implicit Q-Learning12 Oct
  2. 2021Conservative Offline Distributional Reinforcement Learning12 Jul
  3. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
  4. 2020MOPO: Model-based Offline Policy Optimization27 May
  5. 2020MOReL : Model-Based Offline Reinforcement Learning12 May
  6. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May
  7. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learning15 Apr

Other references · 22

  1. 2017Attention Is All You Needcited by 59,932
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2013Auto-Encoding Variational Bayescited by 3,980
  5. 2013Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computationcited by 2,031
  6. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,528
  7. 2019Mastering Atari, Go, Chess and Shogi by Planning with a Learned Modelcited by 1,327
  8. 2020Mastering Atari with Discrete World Modelscited by 990
  9. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 750
  10. 2021Offline Reinforcement Learning as One Big Sequence Modeling Problemcited by 636
  11. 2021Multi-Modal Fusion Transformer for End-to-End Autonomous Drivingcited by 372
  12. 2018Implicit Quantile Networks for Distributional Reinforcement Learningcited by 362
10 more
  1. 2019Exploring the Limitations of Behavior Cloning for Autonomous Drivingcited by 343
  2. 2017Revisiting the Arcade Learning Environment: Evaluation Protocols and Open Problems for General Agentscited by 267
  3. 2021Multimodal Motion Prediction with Stacked Transformerscited by 207
  4. 2019Multiple Futures Predictioncited by 178
  5. 2021Scene Transformer: A unified architecture for predicting multiple agent trajectoriescited by 165
  6. 2013Risk-sensitive Reinforcement Learningcited by 129
  7. 2021Learning to drive from a world on railscited by 114
  8. 2021Vector Quantized Models for Planningcited by 47
  9. 2021Predicting Vehicles Trajectories in Urban Scenarios with Transformer Networks and Augmented Informationcited by 9
  10. 2018Attention-based Recurrent Neural Network for Urban Vehicle Trajectory Predictioncited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 3; 2023: 9; 2024: 4; 2025: 2; 2026: 4.

22 papers.

Sort:

By area:

  1. 2022Is Conditional Generative Modeling all you need for Decision-Making?cited by 504
  2. 2023HIQL: Offline Goal-Conditioned RL with Latent States as Actionscited by 133
  3. 2023Reinforcement Learning from Passive Data via Latent Intentionscited by 64
  4. 2023A Survey on Transformers in Reinforcement Learningcited by 50
  5. 2022Dichotomy of Control: Separating What You Can Control from What You Cannotcited by 47
  6. 2022On Transforming Reinforcement Learning by Transformer: The Development Trajectorycited by 35
  7. 2023Future-conditioned Unsupervised Pretraining for Decision Transformercited by 26
  8. 2023ACT: Empowering Decision Transformer with Dynamic Programming via Advantage Conditioningcited by 24
  9. 2023Graph Decision Transformercited by 20
  10. 2024Deep Generative Models for Offline Policy Learning: Tutorial, Survey, and Perspectives on Future Directionscited by 20
  11. 2025YESciEval: Robust LLM-as-a-Judge for Scientific Question Answeringcited by 15
  12. 2024Latent Plan Transformer for Trajectory Abstraction: Planning as Latent Space Inferencecited by 14
10 more
  1. 2023Uncertainty-Aware Decision Transformer for Stochastic Driving Environmentscited by 8
  2. 2025Structural Information-based Hierarchical Diffusion for Offline Reinforcement Learningcited by 3
  3. 2024DuoSpaceNet: Leveraging Both Bird's-Eye-View and Perspective View Representations for 3D Object Detectioncited by 2
  4. 2026Abstraction for Offline Goal-Conditioned Reinforcement Learningcited by 2
  5. 2026Benchmarking the Limits of In-Context Reinforcement Learning for Ad-Hoc Teamworkcited by 2
  6. 2023Deep Spectral Q-learning with Application to Mobile Healthcited by 0
  7. 2023Efficiency Separation between RL Methods: Model-Free, Model-Based and Goal-Conditionedcited by 0
  8. 2024Decision Mamba Architecturescited by 0
  9. 2026Cost-Aware Diffusion Active Searchcited by 0
  10. 2026Transformer-Enhanced Reinforcement Learning: Fundamentals and Applications in Communication Networkscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025AD-R1: Closed-Loop Reinforcement Learning for End-to-End Autonomous Driving with Impartial World Models
  2. 2026Optimistic World Models: Efficient Exploration in Model-Based Deep Reinforcement Learning
  3. 2024Tackling Data Corruption in Offline Reinforcement Learning via Sequence Modeling
  4. 2021Pessimistic Model Selection for Offline Deep Reinforcement Learning
  5. 2022Model-Based Offline Reinforcement Learning with Pessimism-Modulated Dynamics Belief
  6. 2026Optimistic Policy Regularization
  7. 2024Optimistic Model Rollouts for Pessimistic Offline Policy Optimization
  8. 2026Dreamer-SAC: Off-Policy Learning in Latent World Models for Sample-Efficient Autonomous Driving