mlsanityData to 14 Oct 2026

A Tractable Inference Perspective of Offline RL

arXiv 2311.00094 ↗First posted 31 Oct 2023cs.LG cs.AIcited by 2

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

A popular paradigm for offline Reinforcement Learning (RL) tasks is to first fit the offline trajectories to a sequence model, and then prompt the model for actions that lead to high expected return. In addition to obtaining accurate sequence models, this paper highlights that tractability, the ability to exactly and efficiently answer various probabilistic queries, plays an important role in offline RL. Specifically, due to the fundamental stochasticity from the offline data-collection policies and the environment dynamics, highly non-trivial conditional/constrained generation is required to elicit rewarding actions. it is still possible to approximate such queries, we observe that such crude estimates significantly undermine the benefits brought by expressive sequence models. To overcome this problem, this paper proposes Trifle (Tractable Inference for Offline RL), which leverages modern Tractable Probabilistic Models (TPMs) to bridge the gap between good sequence models and high expected returns at evaluation time. Empirically, Trifle achieves the most state-of-the-art scores in 9 Gym-MuJoCo benchmarks against strong baselines. Further, owing to its tractability, Trifle significantly outperforms prior approaches in stochastic environments and safe RL tasks (e.g. with action constraints) with minimum algorithmic modifications.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 37 of its 50 references that are in the map. First posted on arXiv 31 Oct 2023.

Builds on

37 of 50 references.

Closest ancestors · 7

Same direction, newest first

  1. 2022When does return-conditioned supervised learning work for offline reinforcement learning?2 Jun
  2. 2022RAMBO-RL: Robust Adversarial Model-Based Offline Reinforcement Learning26 Apr
  3. 2021RvS: What is Essential for Offline RL via Supervised Learning?20 Dec
  4. 2021Offline Reinforcement Learning with Implicit Q-Learning12 Oct
  5. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
  6. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May
  7. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learning15 Apr

Other references · 30

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  4. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,546
  5. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,241
  6. 2018Soft Actor-Critic Algorithms and Applicationscited by 1,240
  7. 2021Variational Diffusion Modelscited by 1,153
  8. 2018Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Modelscited by 774
  9. 2021Offline Reinforcement Learning as One Big Sequence Modeling Problemcited by 644
  10. 2020NVAE: A Deep Hierarchical Variational Autoencodercited by 637
  11. 2022Is Conditional Generative Modeling all you need for Decision-Making?cited by 516
  12. 2019Model-Based Reinforcement Learning for Ataricited by 502
18 more
  1. 2022Online Decision Transformercited by 196
  2. 2015Learning Continuous Control Policies by Stochastic Value Gradientscited by 162
  3. 2019Reinforcement Learning Upside Down: Don't Predict Rewards -- Just Map Them to Actionscited by 125
  4. 2019Training Agents using Upside-Down Reinforcement Learningcited by 107
  5. 2022Q-learning Decision Transformer: Leveraging Dynamic Programming for Conditional Sequence Modelling in Offline RLcited by 103
  6. 2019Reward-Conditioned Policiescited by 102
  7. 2022You Can't Count on Luck: Why Decision Transformers and RvS Fail in Stochastic Environmentscited by 63
  8. 2020On the model-based stochastic value gradient for continuous reinforcement learningcited by 47
  9. 2022Dichotomy of Control: Separating What You Can Control from What You Cannotcited by 47
  10. 2016Self-Correcting Models for Model-Based Reinforcement Learningcited by 43
  11. 2016On the Latent Variable Interpretation in Sum-Product Networkscited by 40
  12. 2021Tractable Regularization of Probabilistic Circuitscited by 39
  13. 2022Scaling Up Probabilistic Circuits by Latent Variable Distillationcited by 35
  14. 2020Mitigating Value Hallucination in Dyna Planning via Multistep Predecessor Modelscited by 28
  15. 2021A Consciousness-Inspired Planning Agent for Model-Based Reinforcement Learningcited by 27
  16. 2022Continuous Mixtures of Tractable Probabilistic Modelscited by 19
  17. 2023Understanding the Distillation Process from Deep Generative Models to Tractable Probabilistic Circuitscited by 19
  18. 2023Bayesian Reparameterization of Reward-Conditioned Reinforcement Learning with Energy-based Modelscited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 1.

2 papers.

  1. 2025TRACE Back from the Future: A Probabilistic Reasoning Approach to Controllable Language Generationcited by 8
  2. 2026The Expressivity Boundary of Probabilistic Circuits: A Comparison with Large Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025A Tutorial: An Intuitive Explanation of Offline Reinforcement Learning Theory
  2. 2025A Clean Slate for Offline Reinforcement Learning
  3. 2020MOReL : Model-Based Offline Reinforcement Learning
  4. 2026STO-RL: Offline RL under Sparse Rewards via LLM-Guided Subgoal Temporal Order
  5. 2025On the Statistical Complexity for Offline and Low-Adaptive Reinforcement Learning with Structures
  6. 2025Fully Offline Reinforcement Learning
  7. 2021A Minimalist Approach to Offline Reinforcement Learning
  8. 2025Model-Based Offline Reinforcement Learning with Reliability-Guaranteed Sequence Modeling