mlsanityData to 7 Oct 2026

Distance Weighted Supervised Learning for Offline Interaction Data

arXiv 2304.13774 ↗First posted 26 Apr 2023cs.LGcited by 18

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

Sequential decision making algorithms often struggle to leverage different sources of unstructured offline interaction data. Imitation learning (IL) methods based on supervised learning are robust, but require optimal demonstrations, which are hard to collect. Offline goal-conditioned reinforcement learning (RL) algorithms promise to learn from sub-optimal data, but face optimization challenges especially with high-dimensional data. To bridge the gap between IL and RL, we introduce Distance Weighted Supervised Learning or DWSL, a supervised method for learning goal-conditioned policies from offline data. DWSL models the entire distribution of time-steps between states in offline data with only supervised learning, and uses this distribution to approximate shortest path distances. To extract a policy, we weight actions by their reduction in distance estimates. Theoretically, DWSL converges to an optimal policy constrained to the data distribution, an attractive property for offline learning, without any bootstrapping. Across all datasets we test, DWSL empirically maintains behavior cloning as a lower bound while still exhibiting policy improvement. In high-dimensional image domains, DWSL surpasses the performance of both prior goal-conditioned IL and RL algorithms. Visualizations and code can be found at https://sites.google.com/view/dwsl/home .

Key facts. Cited by 18 papers in the map (data through 7 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 43 of its 57 references that are in the map. First posted on arXiv 26 Apr 2023.

Builds on

43 of 57 references.

Closest ancestors · 7

Same direction, newest first

  1. 2022Don't Change the Algorithm, Change the Data: Exploratory Data for Offline Reinforcement Learning31 Jan
  2. 2021RvS: What is Essential for Offline RL via Supervised Learning?20 Dec
  3. 2021Offline Reinforcement Learning with Implicit Q-Learning12 Oct
  4. 2021A Minimalist Approach to Offline Reinforcement Learning12 Jun
  5. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
  6. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learning15 Apr
  7. 2019Behavior Regularized Offline Reinforcement Learning26 Nov

Other references · 36

  1. 2015Trust Region Policy Optimizationcited by 2,503
  2. 2022RT-1: Robotics Transformer for Real-World Control at Scalecited by 2,007
  3. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,528
  4. 2017Hindsight Experience Replaycited by 1,096
  5. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 750
  6. 2021What Matters in Learning from Offline Human Demonstrations for Robot Manipulationcited by 699
  7. 2017Reinforcement Learning with Deep Energy-Based Policiescited by 659
  8. 2020AWAC: Accelerating Online Reinforcement Learning with Offline Datasetscited by 647
  9. 2021Offline Reinforcement Learning as One Big Sequence Modeling Problemcited by 636
  10. 2022BC-Z: Zero-Shot Task Generalization with Robotic Imitation Learningcited by 517
  11. 2019Relay Policy Learning: Solving Long-Horizon Tasks via Imitation and Reinforcement Learningcited by 380
  12. 2019Learning Latent Plans from Playcited by 310
24 more
  1. 2019Improving Sample Efficiency in Model-Free Reinforcement Learning from Imagescited by 268
  2. 2018Multi-Goal Reinforcement Learning: Challenging Robotics Environments and Request for Researchcited by 256
  3. 2017One-Shot Imitation Learningcited by 232
  4. 2020Accelerating Reinforcement Learning with Learned Skill Priorscited by 220
  5. 2019Search on the Replay Buffer: Bridging Planning and Reinforcement Learningcited by 188
  6. 2018Deep Reinforcement Learning and the Deadly Triadcited by 170
  7. 2019Goal-conditioned Imitation Learningcited by 159
  8. 2015Deep Spatial Autoencoders for Visuomotor Learningcited by 149
  9. 2021Actionable Models: Unsupervised Offline Reinforcement Learning of Robotic Skillscited by 125
  10. 2019Training Agents using Upside-Down Reinforcement Learningcited by 106
  11. 2022From Play to Policy: Conditional Behavior Generation from Uncurated Robot Datacited by 103
  12. 2020C-Learning: Learning to Achieve Goals via Recursive Classificationcited by 99
  13. 2022Rethinking Goal-conditioned Supervised Learning and Its Connection to Offline RLcited by 89
  14. 2022Latent Plans for Task-Agnostic Offline Reinforcement Learningcited by 81
  15. 2019Dynamical Distance Learning for Semi-Supervised and Unsupervised Skill Discoverycited by 65
  16. 2022How Far I'll Go: Offline Goal-Conditioned Reinforcement Learning via f-Advantage Regressioncited by 65
  17. 2020Rewriting History with Inverse RL: Hindsight Inference for Policy Improvementcited by 59
  18. 2020Generalized Hindsight for Reinforcement Learningcited by 51
  19. 2022Imitation Learning by Estimating Expertise of Demonstratorscited by 50
  20. 2021Confidence-Aware Imitation Learning from Demonstrations with Varying Optimalitycited by 44
  21. 2020Model-Based Visual Planning with Self-Supervised Functional Distancescited by 41
  22. 2021Learning from Imperfect Demonstrations from Agents with Varying Dynamicscited by 26
  23. 2022Generalization with Lossy Affordances: Leveraging Broad Offline Data for Learning Visuomotor Taskscited by 24
  24. 2022PLATO: Predicting Latent Affordances Through Object-Centric Playcited by 12

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 4; 2024: 5; 2025: 6; 2026: 3.

18 papers.

Sort:

By area:

  1. 2024OGBench: Benchmarking Offline Goal-Conditioned RLcited by 216
  2. 2023Contrastive Preference Learning: Learning from Human Feedback without RLcited by 77
  3. 2024Learning Temporal Distances: Contrastive Successor Features Can Provide a Metric Structure for Decision-Makingcited by 53
  4. 2025Self-Improving Embodied Foundation Modelscited by 28
  5. 2023ACT: Empowering Decision Transformer with Dynamic Programming via Advantage Conditioningcited by 24
  6. 2023SMORE: Score Models for Offline Goal-Conditioned Reinforcement Learningcited by 21
  7. 2024Inference via Interpolation: Contrastive Representations Provably Enable Planning and Inferencecited by 15
  8. 2025Transitive RL: Value Learning via Divide and Conquercited by 12
  9. 2023Goal-Conditioned Predictive Coding for Offline Reinforcement Learningcited by 10
  10. 2025Penalizing Infeasible Actions and Reward Scaling in Reinforcement Learning with Offline Datacited by 7
  11. 2024Offline Goal-Conditioned Reinforcement Learning for Safety-Critical Tasks with Recovery Policycited by 5
  12. 2025Temporal Distance-aware Transition Augmentation for Offline Model-based Reinforcement Learningcited by 5
6 more
  1. 2026Seeing Farther and Smarter: Value-Guided Multi-Path Reflection for VLM Policy Optimizationcited by 4
  2. 2026SVL: Goal-Conditioned Reinforcement Learning as Survival Learningcited by 3
  3. 2025GCHR : Goal-Conditioned Hindsight Regularization for Sample-Efficient Reinforcement Learningcited by 2
  4. 2024MGDA: Model-based Goal Data Augmentation for Offline Goal-conditioned Weighted Supervised Learningcited by 1
  5. 2026Occupancy Reward Shaping: Improving Credit Assignment for Offline Goal-Conditioned Reinforcement Learningcited by 1
  6. 2025Reachability Weighted Offline Goal-conditioned Resamplingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2022Discriminator-Weighted Offline Imitation Learning from Suboptimal Demonstrations
  2. 2021Representation Matters: Offline Pretraining for Sequential Decision Making
  3. 2021Bridging Offline Reinforcement Learning and Imitation Learning: A Tale of Pessimism
  4. 2023Offline Imitation Learning with Suboptimal Demonstrations via Relaxed Distribution Matching
  5. 2023Using Offline Data to Speed Up Reinforcement Learning in Procedurally Generated Environments
  6. 2025Using Non-Expert Data to Robustify Imitation Learning via Offline Reinforcement Learning
  7. 2023Harnessing Mixed Offline Reinforcement Learning Datasets via Trajectory Weighting
  8. 2023Optimal Transport for Offline Imitation Learning