mlsanityData to 14 Oct 2026

Learning Value Functions from Undirected State-only Experience

arXiv 2204.12458 ↗First posted 26 Apr 2022cs.LG cs.AI cs.ROcited by 8

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

This paper tackles the problem of learning value functions from undirected state-only experience (state transitions without action labels i.e. (s,s',r) tuples). We first theoretically characterize the applicability of Q-learning in this setting. We show that tabular Q-learning in discrete Markov decision processes (MDPs) learns the same value function under any arbitrary refinement of the action space. This theoretical result motivates the design of Latent Action Q-learning or LAQ, an offline RL method that can learn effective value functions from state-only experience. Latent Action Q-learning (LAQ) learns value functions using Q-learning on discrete latent actions obtained through a latent-variable future prediction model. We show that LAQ can recover value functions that have high correlation with value functions learned using ground truth actions. Value functions learned using LAQ lead to sample efficient acquisition of goal-directed behavior, can be used with domain-specific low-level controllers, and facilitate transfer across embodiments. Our experiments in 5 environments ranging from 2D grid world to 3D visual navigation in realistic environments demonstrate the benefits of LAQ over simpler alternatives, imitation learning oracles, and competing methods.

Key facts. Cited by 8 papers in the map (data through 14 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 33 of its 56 references that are in the map. First posted on arXiv 26 Apr 2022.

Builds on

33 of 56 references.

Closest ancestors · 5

Same direction, newest first

  1. 2021A Workflow for Offline Model-Free Robotic Reinforcement Learning22 Sep
  2. 2020Offline Reinforcement Learning from Images with Latent Space Models21 Dec
  3. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
  4. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May
  5. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learning15 Apr

Other references · 28

  1. 2013Playing Atari with Deep Reinforcement Learningcited by 3,274
  2. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,241
  3. 2019Habitat: A Platform for Embodied AI Researchcited by 1,191
  4. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 840
  5. 2018On Evaluation of Embodied Navigation Agentscited by 547
  6. 2019Meta-Learning with Implicit Gradientscited by 525
  7. 2018Behavioral Cloning from Observationcited by 492
  8. 2016Unsupervised Learning for Physical Interaction through Video Predictioncited by 260
  9. 2018Generative Adversarial Imitation from Observationcited by 184
  10. 2015Action-Conditional Video Prediction using Deep Networks in Atari Gamescited by 172
  11. 2016Learning to Poke by Poking: Experiential Learning of Intuitive Physicscited by 154
  12. 2020Acme: A Research Framework for Distributed Reinforcement Learningcited by 146
16 more
  1. 2020Model-Based Offline Planningcited by 129
  2. 2019End-to-End Robotic Reinforcement Learning without Reward Engineeringcited by 122
  3. 2019Grasping in the Wild:Learning 6DoF Closed-Loop Grasping from Low-Cost Demonstrationscited by 107
  4. 2019Improved Conditional VRNNs for Video Predictioncited by 104
  5. 2018Imitating Latent Policies from Observationcited by 101
  6. 2020State-Only Imitation Learning for Dexterous Manipulationcited by 91
  7. 2019IRIS: Implicit Reinforcement without Interaction at Scale for Learning Control from Offline Robot Manipulation Datacited by 86
  8. 2016Unsupervised Perceptual Rewards for Imitation Learningcited by 65
  9. 2020Plannable Approximations to MDP Homomorphisms: Equivariance under Actionscited by 61
  10. 2020Semantic Visual Navigation by Watching YouTube Videoscited by 60
  11. 2016Visual Dynamics: Probabilistic Future Frame Synthesis via Cross Convolutional Networkscited by 46
  12. 2018Few-Shot Goal Inference for Visuomotor Learning and Planningcited by 30
  13. 2019Imitation Learning from Video by Leveraging Proprioceptioncited by 23
  14. 2020Estimating Q(s,s') with Deep Deterministic Dynamics Gradientscited by 12
  15. 2019Perceptual Values from Observationcited by 11
  16. 2019Learning Navigation Subroutines from Egocentric Videoscited by 7

Built on by

Cited by year

Citing papers per year: 2022: 1; 2023: 4; 2024: 2; 2025: 1.

8 papers.

  1. 2023HIQL: Offline Goal-Conditioned RL with Latent States as Actionscited by 141
  2. 2023Reinforcement Learning from Passive Data via Latent Intentionscited by 68
  3. 2024Towards Generalist Robot Learning from Internet Video: A Surveycited by 40
  4. 2023Robotic Offline RL from Internet Videos via Value-Function Pre-Trainingcited by 33
  5. 2023Look Ma, No Hands! Agent-Environment Factorization of Egocentric Videoscited by 21
  6. 2024Pre-trained Visual Dynamics Representations for Efficient Policy Learningcited by 8
  7. 2022State Advantage Weighting for Offline RLcited by 5
  8. 2025Video-Enhanced Offline Reinforcement Learning: A Model-Based Approachcited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2026Long-Horizon Q-Learning: Accurate Value Learning via n-Step Inequalities
  2. 2024Time-Scale Separation in Q-Learning: Extending TD(triangle) for Action-Value Function Decomposition
  3. 2025Learning from Suboptimal Data in Continuous Control via Auto-Regressive Soft Q-Network
  4. 2025An Analysis of Action-Value Temporal-Difference Methods That Learn State Values
  5. 2025Actor-Free Continuous Control via Structurally Maximizable Q-Functions
  6. 2025Local Reinforcement Learning with Action-Conditioned Root Mean Squared Q-Functions
  7. 2024Stochastic Q-learning for Large Discrete Action Spaces
  8. 2025Beyond Shallow Behavior: Task-Efficient Value-Based Multi-Task Offline MARL via Skill Discovery