mlsanityData to 7 Oct 2026

Flow to Control: Offline Reinforcement Learning with Lossless Primitive Discovery

arXiv 2212.01105 ↗First posted 2 Dec 2022cs.LG cs.AIcited by 12

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

Offline reinforcement learning (RL) enables the agent to effectively learn from logged data, which significantly extends the applicability of RL algorithms in real-world scenarios where exploration can be expensive or unsafe. Previous works have shown that extracting primitive skills from the recurring and temporally extended structures in the logged data yields better learning. However, these methods suffer greatly when the primitives have limited representation ability to recover the original policy space, especially in offline settings. In this paper, we give a quantitative characterization of the performance of offline hierarchical learning and highlight the importance of learning lossless primitives. To this end, we propose to use a flow-based structure as the representation for low-level policies. This allows us to represent the behaviors in the dataset faithfully while keeping the expression ability to recover the whole policy space. We show that such lossless primitives can drastically improve the performance of hierarchical policies. The experimental results and extensive ablation studies on the standard D4RL benchmark show that our method has a good representation ability for policies and achieves superior performance in most tasks.

Key facts. Cited by 12 papers in the map (data through 7 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 36 of its 40 references that are in the map. First posted on arXiv 2 Dec 2022.

Builds on

36 of 40 references.

Closest ancestors · 15

Same direction, newest first

  1. 2022On the Role of Discount Factor in Offline Reinforcement Learning7 Jun
  2. 2021Offline Reinforcement Learning with Value-based Episodic Memory19 Oct
  3. 2021Offline Reinforcement Learning with Implicit Q-Learning12 Oct
  4. 2021Pessimistic Model-based Offline Reinforcement Learning under Partial Coverage13 Jul
  5. 2021Conservative Offline Distributional Reinforcement Learning12 Jul
  6. 2021Offline RL Without Off-Policy Evaluation16 Jun
  7. 2021A Minimalist Approach to Offline Reinforcement Learning12 Jun
  8. 2021COMBO: Conservative Offline Model-Based Policy Optimization16 Feb
  9. 2020Is Pessimism Provably Efficient for Offline RL?30 Dec
  10. 2020OPAL: Offline Primitive Discovery for Accelerating Offline Reinforcement Learning26 Oct
  11. 2020EMaQ: Expected-Max Q-Learning Operator for Simple Yet Effective Offline and Online RL21 Jul
  12. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
3 more
  1. 2020MOReL : Model-Based Offline Reinforcement Learning12 May
  2. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May
  3. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learning15 Apr

Other references · 21

  1. 2015Continuous control with deep reinforcement learningcited by 3,279
  2. 2016Density estimation using Real NVPcited by 2,029
  3. 2014NICE: Non-linear Independent Components Estimationcited by 1,052
  4. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 750
  5. 2018Diversity is All You Need: Learning Skills without a Reward Functioncited by 734
  6. 2020AWAC: Accelerating Online Reinforcement Learning with Offline Datasetscited by 647
  7. 2019Provably Efficient Reinforcement Learning with Linear Function Approximationcited by 514
  8. 2017Distributional Reinforcement Learning with Quantile Regressioncited by 464
  9. 2019Relay Policy Learning: Solving Long-Horizon Tasks via Imitation and Reinforcement Learningcited by 380
  10. 2019Dynamics-Aware Unsupervised Discovery of Skillscited by 308
  11. 2016Safe, Multi-Agent, Reinforcement Learning for Autonomous Drivingcited by 301
  12. 2016Hierarchical Deep Reinforcement Learning: Integrating Temporal Abstraction and Intrinsic Motivationcited by 282
9 more
  1. 2020Accelerating Reinforcement Learning with Learned Skill Priorscited by 220
  2. 2019Mastering Complex Control in MOBA Games with Deep Reinforcement Learningcited by 150
  3. 2020Parrot: Data-Driven Behavioral Priors for Reinforcement Learningcited by 136
  4. 2020Model-Based Offline Planningcited by 126
  5. 2018Near-Optimal Representation Learning for Hierarchical Reinforcement Learningcited by 107
  6. 2020COG: Connecting New Skills to Past Experience with Offline Reinforcement Learningcited by 83
  7. 2021Believe What You See: Implicit Constraint Approach for Offline Multi-Agent Reinforcement Learningcited by 78
  8. 2020Learning Robot Skills with Temporal Variational Inferencecited by 54
  9. 2022Skill-based Meta-Reinforcement Learningcited by 36

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 4; 2024: 4; 2025: 2; 2026: 2.

12 papers.

  1. 2023Reasoning with Latent Diffusion in Offline Reinforcement Learningcited by 33
  2. 2024Bayesian Design Principles for Offline-to-Online Reinforcement Learningcited by 14
  3. 2023Unsupervised Behavior Extraction via Random Intent Priorscited by 13
  4. 2023Efficient Planning with Latent Diffusioncited by 11
  5. 2023The Provable Benefits of Unsupervised Data Sharing for Offline Reinforcement Learningcited by 10
  6. 2025Fewer May Be Better: Enhancing Offline Reinforcement Learning with Reduced Datasetcited by 4
  7. 2024Disentangling Policy from Offline Task Representation Learning via Adversarial Data Augmentationcited by 2
  8. 2024DIAR: Diffusion-model-guided Implicit Q-learning with Adaptive Revaluationcited by 2
  9. 2024Navigation with QPHIL: Quantizing Planner for Hierarchical Implicit Q-Learningcited by 2
  10. 2025Decision Flow Policy Optimizationcited by 0
  11. 2026OPRIDE: Offline Preference-based Reinforcement Learning via In-Dataset Explorationcited by 0
  12. 2026Sim2O: Efficient Offline-to-Online MARL via Joint Action Compositioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Offline Hierarchical Reinforcement Learning via Inverse Optimization
  2. 2025A Tutorial: An Intuitive Explanation of Offline Reinforcement Learning Theory
  3. 2021A Workflow for Offline Model-Free Robotic Reinforcement Learning
  4. 2026Flow Actor-Critic for Offline Reinforcement Learning
  5. 2021Representation Matters: Offline Pretraining for Sequential Decision Making
  6. 2023Beyond Uniform Sampling: Offline Reinforcement Learning with Imbalanced Datasets
  7. 2022Behavior Prior Representation learning for Offline Reinforcement Learning
  8. 2026Efficient Hierarchical Implicit Flow Q-learning for Offline Goal-conditioned Reinforcement Learning