mlsanityData to 14 Oct 2026

CLUE: Calibrated Latent Guidance for Offline Reinforcement Learning

arXiv 2306.13412 ↗First posted 23 Jun 2023cs.LGcited by 11

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

Offline reinforcement learning (RL) aims to learn an optimal policy from pre-collected and labeled datasets, which eliminates the time-consuming data collection in online RL. However, offline RL still bears a large burden of specifying/handcrafting extrinsic rewards for each transition in the offline data. As a remedy for the labor-intensive labeling, we propose to endow offline RL tasks with a few expert data and utilize the limited expert data to drive intrinsic rewards, thus eliminating the need for extrinsic rewards. To achieve that, we introduce Calibrated Latent gUidancE (CLUE), which utilizes a conditional variational auto-encoder to learn a latent space such that intrinsic rewards can be directly qualified over the latent space. CLUE's key idea is to align the intrinsic rewards consistent with the expert intention via enforcing the embeddings of expert data to a calibrated contextual representation. We instantiate the expert-driven intrinsic rewards in sparse-reward offline RL tasks, offline imitation learning (IL) tasks, and unsupervised offline RL tasks. Empirically, we find that CLUE can effectively improve the sparse-reward offline RL performance, outperform the state-of-the-art offline IL baselines, and discover diverse skills from static reward-free offline data.

Key facts. Cited by 11 papers in the map (data through 14 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 48 of its 69 references that are in the map. First posted on arXiv 23 Jun 2023.

Builds on

48 of 69 references.

Closest ancestors · 9

Same direction, newest first

  1. 2023Beyond OOD State Actions: Supported Cross-Domain Offline Reinforcement Learning22 Jun
  2. 2023Beyond Reward: Offline Preference-guided Policy Optimization25 May
  3. 2023Optimal Transport for Offline Imitation Learning24 Mar
  4. 2023Behavior Proximal Policy Optimization22 Feb
  5. 2022DARA: Dynamics-Aware Reward Augmentation in Offline Reinforcement Learning13 Mar
  6. 2022Supported Policy Optimization for Offline Reinforcement Learning13 Feb
  7. 2021Offline Reinforcement Learning with Implicit Q-Learning12 Oct
  8. 2021A Minimalist Approach to Offline Reinforcement Learning12 Jun
  9. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May

Other references · 39

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2017Curiosity-driven Exploration by Self-supervised Predictioncited by 1,194
  3. 2018Exploration by Random Network Distillationcited by 993
  4. 2018Diversity is All You Need: Learning Skills without a Reward Functioncited by 752
  5. 2018QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulationcited by 743
  6. 2016Unifying Count-Based Exploration and Intrinsic Motivationcited by 639
  7. 2020Planning to Explore via Self-Supervised World Modelscited by 368
  8. 2018Reward learning from human preferences and demonstrations in Ataricited by 316
  9. 2019Dynamics-Aware Unsupervised Discovery of Skillscited by 313
  10. 2016#Exploration: A Study of Count-Based Exploration for Deep Reinforcement Learningcited by 309
  11. 2019Self-Supervised Exploration via Disagreementcited by 265
  12. 2017Count-Based Exploration with Neural Density Modelscited by 257
27 more
  1. 2020Never Give Up: Learning Directed Exploration Strategiescited by 212
  2. 2021IQ-Learn: Inverse soft-Q Learning for Imitationcited by 193
  3. 2019Efficient Exploration via State Marginal Matchingcited by 188
  4. 2019Imitation Learning via Off-Policy Distribution Matchingcited by 183
  5. 2021Behavior From the Void: Unsupervised Active Pre-Trainingcited by 174
  6. 2021URLB: Unsupervised Reinforcement Learning Benchmarkcited by 139
  7. 2019End-to-End Robotic Reinforcement Learning without Reward Engineeringcited by 122
  8. 2020Explore, Discover and Learn: Unsupervised Discovery of State-Covering Skillscited by 122
  9. 2021Rapid Exploration for Open-World Navigation with Latent Goal Modelscited by 121
  10. 2021State Entropy Maximization with Random Encoders for Efficient Explorationcited by 108
  11. 2022Discriminator-Weighted Offline Imitation Learning from Suboptimal Demonstrationscited by 89
  12. 2021Of Moments and Matching: A Game-Theoretic Framework for Closing the Imitation Gapcited by 81
  13. 2019InfoBot: Transfer and Exploration via the Information Bottleneckcited by 78
  14. 2020Offline Learning from Demonstrations and Unlabeled Experiencecited by 72
  15. 2020Learning Reward Functions from Diverse Sources of Human Feedback: Optimally Integrating Demonstrations and Preferencescited by 71
  16. 2013Empowerment -- an Introductioncited by 68
  17. 2019What Can Learned Intrinsic Rewards Capture?cited by 36
  18. 2020Intrinsic Reward Driven Imitation Learning via Generative Modelcited by 35
  19. 2019Deep Bayesian Reward Learning from Preferencescited by 28
  20. 2021Information Directed Reward Learning for Reinforcement Learningcited by 24
  21. 2021Learn Goal-Conditioned Policy with Intrinsic Motivation for Deep Reinforcement Learningcited by 17
  22. 2021Unsupervised Domain Adaptation with Dynamics-Aware Rewards in Reinforcement Learningcited by 15
  23. 2021SoftDICE for Imitation Learning: Rethinking Off-policy Distribution Matchingcited by 14
  24. 2021Offline Inverse Reinforcement Learningcited by 14
  25. 2022Learn what matters: cross-domain imitation learning with task-relevant embeddingscited by 14
  26. 2018Learning to Share and Hide Intentions using Information Regularizationcited by 13
  27. 2019SMiRL: Surprise Minimizing Reinforcement Learning in Unstable Environmentscited by 9

Built on by

Cited by year

Citing papers per year: 2023: 1; 2024: 6; 2025: 4.

11 papers.

  1. 2025A Survey on Imitation Learning for Contact-Rich Tasks in Roboticscited by 14
  2. 2024Align Your Intents: Offline Imitation Learning via Optimal Transportcited by 8
  3. 2023Offline Imitation Learning with Variational Counterfactual Reasoningcited by 7
  4. 2024DIDI: Diffusion-Guided Diversity for Offline Behavioral Generationcited by 4
  5. 2024Discovering Multiple Solutions from a Single Task in Offline Reinforcement Learningcited by 4
  6. 2024Context-Former: Stitching via Latent Conditioned Sequence Modelingcited by 3
  7. 2025Dual-Force: Enhanced Offline Diversity Maximization under Imitation Constraintscited by 2
  8. 2024Dataset Clustering for Improved Offline Policy Learningcited by 1
  9. 2025From Novelty to Imitation: Self-Distilled Rewards for Offline Reinforcement Learningcited by 1
  10. 2024Imitating from auxiliary imperfect demonstrations via Adversarial Density Weighted Regressioncited by 0
  11. 2025Minimal Ingredients for Reward Assignment from Expert Demonstrationscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Reasoning with Latent Diffusion in Offline Reinforcement Learning
  2. 2026STO-RL: Offline RL under Sparse Rewards via LLM-Guided Subgoal Temporal Order
  3. 2023Cal-QL: Calibrated Offline RL Pre-Training for Efficient Online Fine-Tuning
  4. 2026OGER: A Robust Offline-Guided Exploration Reward for Hybrid Reinforcement Learning
  5. 2025Learning from Reward-Free Offline Data: A Case for Planning with Latent Dynamics Models
  6. 2021Offline Reinforcement Learning as Anti-Exploration
  7. 2025Expressive Value Learning for Scalable Offline Reinforcement Learning
  8. 2025A Clean Slate for Offline Reinforcement Learning