# Distance Weighted Supervised Learning for Offline Interaction Data

arXiv: https://arxiv.org/abs/2304.13774 · HTML: https://mlsanity.com/paper/2304.13774/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data.
- Builds on 43 of its 57 references that are in the map.
- First posted on arXiv 26 Apr 2023.

## Placed in

- Offline RL and imitation learning from data / [Offline RL algorithms](https://mlsanity.com/offlinerl/offline-rl-algorithms/)

## Builds on (most cited first, 30 of 43)

- [Trust Region Policy Optimization](https://mlsanity.com/paper/1502.05477/) (2015)
- [RT-1: Robotics Transformer for Real-World Control at Scale](https://mlsanity.com/paper/2212.06817/) (2022)
- [Conservative Q-Learning for Offline Reinforcement Learning](https://mlsanity.com/paper/2006.04779/) (2020)
- [Decision Transformer: Reinforcement Learning via Sequence Modeling](https://mlsanity.com/paper/2106.01345/) (2021)
- [D4RL: Datasets for Deep Data-Driven Reinforcement Learning](https://mlsanity.com/paper/2004.07219/) (2020)
- [Offline Reinforcement Learning with Implicit Q-Learning](https://mlsanity.com/paper/2110.06169/) (2021)
- [Hindsight Experience Replay](https://mlsanity.com/paper/1707.01495/) (2017)
- [A Minimalist Approach to Offline Reinforcement Learning](https://mlsanity.com/paper/2106.06860/) (2021)
- [Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learning](https://mlsanity.com/paper/1910.00177/) (2019)
- [What Matters in Learning from Offline Human Demonstrations for Robot Manipulation](https://mlsanity.com/paper/2108.03298/) (2021)
- [AWAC: Accelerating Online Reinforcement Learning with Offline Datasets](https://mlsanity.com/paper/2006.09359/) (2020)
- [Reinforcement Learning with Deep Energy-Based Policies](https://mlsanity.com/paper/1702.08165/) (2017)
- [Behavior Regularized Offline Reinforcement Learning](https://mlsanity.com/paper/1911.11361/) (2019)
- [Offline Reinforcement Learning as One Big Sequence Modeling Problem](https://mlsanity.com/paper/2106.02039/) (2021)
- [BC-Z: Zero-Shot Task Generalization with Robotic Imitation Learning](https://mlsanity.com/paper/2202.02005/) (2022)
- [Relay Policy Learning: Solving Long-Horizon Tasks via Imitation and Reinforcement Learning](https://mlsanity.com/paper/1910.11956/) (2019)
- [Learning Latent Plans from Play](https://mlsanity.com/paper/1903.01973/) (2019)
- [Improving Sample Efficiency in Model-Free Reinforcement Learning from Images](https://mlsanity.com/paper/1910.01741/) (2019)
- [Multi-Goal Reinforcement Learning: Challenging Robotics Environments and Request for Research](https://mlsanity.com/paper/1802.09464/) (2018)
- [One-Shot Imitation Learning](https://mlsanity.com/paper/1703.07326/) (2017)
- [Accelerating Reinforcement Learning with Learned Skill Priors](https://mlsanity.com/paper/2010.11944/) (2020)
- [Search on the Replay Buffer: Bridging Planning and Reinforcement Learning](https://mlsanity.com/paper/1906.05253/) (2019)
- [RvS: What is Essential for Offline RL via Supervised Learning?](https://mlsanity.com/paper/2112.10751/) (2021)
- [Deep Reinforcement Learning and the Deadly Triad](https://mlsanity.com/paper/1812.02648/) (2018)
- [Goal-conditioned Imitation Learning](https://mlsanity.com/paper/1906.05838/) (2019)
- [Deep Spatial Autoencoders for Visuomotor Learning](https://mlsanity.com/paper/1509.06113/) (2015)
- [Actionable Models: Unsupervised Offline Reinforcement Learning of Robotic Skills](https://mlsanity.com/paper/2104.07749/) (2021)
- [Training Agents using Upside-Down Reinforcement Learning](https://mlsanity.com/paper/1912.02877/) (2019)
- [C-Learning: Learning to Achieve Goals via Recursive Classification](https://mlsanity.com/paper/2011.08909/) (2020)
- [From Play to Policy: Conditional Behavior Generation from Uncurated Robot Data](https://mlsanity.com/paper/2210.10047/) (2022)

## Built on by (most cited first, 18 of 18)

- [OGBench: Benchmarking Offline Goal-Conditioned RL](https://mlsanity.com/paper/2410.20092/) (2024)
- [Contrastive Preference Learning: Learning from Human Feedback without RL](https://mlsanity.com/paper/2310.13639/) (2023)
- [Learning Temporal Distances: Contrastive Successor Features Can Provide a Metric Structure for Decision-Making](https://mlsanity.com/paper/2406.17098/) (2024)
- [Self-Improving Embodied Foundation Models](https://mlsanity.com/paper/2509.15155/) (2025)
- [ACT: Empowering Decision Transformer with Dynamic Programming via Advantage Conditioning](https://mlsanity.com/paper/2309.05915/) (2023)
- [SMORE: Score Models for Offline Goal-Conditioned Reinforcement Learning](https://mlsanity.com/paper/2311.02013/) (2023)
- [Inference via Interpolation: Contrastive Representations Provably Enable Planning and Inference](https://mlsanity.com/paper/2403.04082/) (2024)
- [Transitive RL: Value Learning via Divide and Conquer](https://mlsanity.com/paper/2510.22512/) (2025)
- [Goal-Conditioned Predictive Coding for Offline Reinforcement Learning](https://mlsanity.com/paper/2307.03406/) (2023)
- [Penalizing Infeasible Actions and Reward Scaling in Reinforcement Learning with Offline Data](https://mlsanity.com/paper/2507.08761/) (2025)
- [Temporal Distance-aware Transition Augmentation for Offline Model-based Reinforcement Learning](https://mlsanity.com/paper/2505.13144/) (2025)
- [Offline Goal-Conditioned Reinforcement Learning for Safety-Critical Tasks with Recovery Policy](https://mlsanity.com/paper/2403.01734/) (2024)
- [Seeing Farther and Smarter: Value-Guided Multi-Path Reflection for VLM Policy Optimization](https://mlsanity.com/paper/2602.19372/) (2026)
- [GCHR : Goal-Conditioned Hindsight Regularization for Sample-Efficient Reinforcement Learning](https://mlsanity.com/paper/2508.06108/) (2025)
- [SVL: Goal-Conditioned Reinforcement Learning as Survival Learning](https://mlsanity.com/paper/2604.17551/) (2026)
- [MGDA: Model-based Goal Data Augmentation for Offline Goal-conditioned Weighted Supervised Learning](https://mlsanity.com/paper/2412.11410/) (2024)
- [Occupancy Reward Shaping: Improving Credit Assignment for Offline Goal-Conditioned Reinforcement Learning](https://mlsanity.com/paper/2604.20627/) (2026)
- [Reachability Weighted Offline Goal-conditioned Resampling](https://mlsanity.com/paper/2506.02577/) (2025)

## Abstract (arXiv)

Sequential decision making algorithms often struggle to leverage different sources of unstructured offline interaction data. Imitation learning (IL) methods based on supervised learning are robust, but require optimal demonstrations, which are hard to collect. Offline goal-conditioned reinforcement learning (RL) algorithms promise to learn from sub-optimal data, but face optimization challenges especially with high-dimensional data. To bridge the gap between IL and RL, we introduce Distance Weighted Supervised Learning or DWSL, a supervised method for learning goal-conditioned policies from offline data. DWSL models the entire distribution of time-steps between states in offline data with only supervised learning, and uses this distribution to approximate shortest path distances. To extract a policy, we weight actions by their reduction in distance estimates. Theoretically, DWSL converges to an optimal policy constrained to the data distribution, an attractive property for offline learning, without any bootstrapping. Across all datasets we test, DWSL empirically maintains behavior cloning as a lower bound while still exhibiting policy improvement. In high-dimensional image domains, DWSL surpasses the performance of both prior goal-conditioned IL and RL algorithms. Visualizations and code can be found at https://sites.google.com/view/dwsl/home .