mlsanityData to 7 Oct 2026

Optimal Transport for Offline Imitation Learning

arXiv 2303.13971 ↗First posted 24 Mar 2023cs.LGcited by 42

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

With the advent of large datasets, offline reinforcement learning (RL) is a promising framework for learning good decision-making policies without the need to interact with the real environment. However, offline RL requires the dataset to be reward-annotated, which presents practical challenges when reward engineering is difficult or when obtaining reward annotations is labor-intensive. In this paper, we introduce Optimal Transport Reward labeling (OTR), an algorithm that assigns rewards to offline trajectories, with a few high-quality demonstrations. OTR's key idea is to use optimal transport to compute an optimal alignment between an unlabeled trajectory in the dataset and an expert demonstration to obtain a similarity measure that can be interpreted as a reward, which can then be used by an offline RL algorithm to learn the policy. OTR is easy to implement and computationally efficient. On D4RL benchmarks, we show that OTR with a single demonstration can consistently match the performance of offline RL with ground-truth rewards.

Key facts. Cited by 42 papers in the map (data through 7 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 19 of its 37 references that are in the map. First posted on arXiv 24 Mar 2023.

Builds on

19 of 37 references.

Closest ancestors · 7

Same direction, newest first

  1. 2022How to Leverage Unlabeled Data in Offline Reinforcement Learning3 Feb
  2. 2021Offline Reinforcement Learning with Implicit Q-Learning12 Oct
  3. 2021A Minimalist Approach to Offline Reinforcement Learning12 Jun
  4. 2020Critic Regularized Regression26 Jun
  5. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
  6. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May
  7. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learning15 Apr

Other references · 12

  1. 2013Sinkhorn Distances: Lightspeed Computation of Optimal Transportation Distancescited by 2,277
  2. 2016Generative Adversarial Imitation Learningcited by 1,576
  3. 2018Computational Optimal Transportcited by 765
  4. 2018Discriminator-Actor-Critic: Addressing Sample Inefficiency and Reward Bias in Adversarial Imitation Learningcited by 203
  5. 2019Imitation Learning via Off-Policy Distribution Matchingcited by 182
  6. 2020Acme: A Research Framework for Distributed Reinforcement Learningcited by 146
  7. 2020Primal Wasserstein Imitation Learningcited by 120
  8. 2022Optimal Transport Tools (OTT): A JAX Toolbox for all things Wassersteincited by 96
  9. 2020Offline Learning from Demonstrations and Unlabeled Experiencecited by 72
  10. 2021What Matters for Adversarial Imitation Learning?cited by 70
  11. 2019Wasserstein Adversarial Imitation Learningcited by 59
  12. 2021Cross-Domain Imitation Learning via Optimal Transportcited by 49

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 9; 2024: 10; 2025: 16; 2026: 7.

42 papers.

Sort:

By area:

  1. 2024A Large Language Model-Driven Reward Design Framework via Dynamic Feedback for Reinforcement Learningcited by 29
  2. 2023What Matters to You? Towards Visual Representation Alignment for Robot Learningcited by 14
  3. 2024SEABO: A Simple Search-Based Method for Offline Imitation Learningcited by 14
  4. 2023CEIL: Generalized Contextual Imitation Learningcited by 12
  5. 2023CLUE: Calibrated Latent Guidance for Offline Reinforcement Learningcited by 11
  6. 2023Imitation Learning from Observation with Automatic Discount Schedulingcited by 10
  7. 2023PEARL: Zero-shot Cross-task Preference Alignment and Robust Reward Learning for Robotic Manipulationcited by 9
  8. 2024Offline Imitation Learning with Model-based Reverse Augmentationcited by 8
  9. 2024Rethinking Optimal Transport in Offline Reinforcement Learningcited by 8
  10. 2024Robot Policy Learning with Temporal Optimal Transport Rewardcited by 8
  11. 2024Align Your Intents: Offline Imitation Learning via Optimal Transportcited by 7
  12. 2025Imitation Learning from a Single Temporally Misaligned Videocited by 7
30 more
  1. 2023Discovering Hierarchical Achievements in Reinforcement Learning via Contrastive Learningcited by 6
  2. 2024ENOT: Expectile Regularization for Fast and Accurate Training of Neural Optimal Transportcited by 6
  3. 2025Direct Post-Training Preference Alignment for Multi-Agent Motion Generation Models Using Implicit Feedback from Pre-training Demonstrationscited by 6
  4. 2023Offline Imitation from Observation via Primal Wasserstein State Occupancy Matchingcited by 5
  5. 2025PINS: Proximal Iterations with Sparse Newton and Sinkhorn for Optimal Transportcited by 5
  6. 2023Leveraging Optimal Transport for Enhanced Offline Reinforcement Learning in Surgical Robotic Environmentscited by 3
  7. 2023A Simple Solution for Offline Imitation from Observations and Examples with Possibly Incomplete Trajectoriescited by 3
  8. 2024Zero-Shot Offline Imitation Learning via Optimal Transportcited by 3
  9. 2024Surgical Task Automation Using Actor-Critic Frameworks and Self-Supervised Imitation Learningcited by 2
  10. 2025PROF: An LLM-based Reward Code Preference Optimization Framework for Offline Imitation Learningcited by 2
  11. 2026Transport and Merge: Cross-Architecture Merging for Large Language Modelscited by 2
  12. 2025DPOT: A DeepParticle method for Computation of Optimal Transport with convergence guaranteecited by 1
  13. 2025Offline Reinforcement Learning with Wasserstein Regularization via Optimal Transport Mapscited by 1
  14. 2025From Novelty to Imitation: Self-Distilled Rewards for Offline Reinforcement Learningcited by 1
  15. 2025Re:Frame -- Retrieving Experience From Associative Memorycited by 1
  16. 2025On Discovering Algorithms for Adversarial Imitation Learningcited by 1
  17. 2026ReGIL: Retrieval-Guided Imitation Learning from a Single Demonstrationcited by 1
  18. 2024Imitating from auxiliary imperfect demonstrations via Adversarial Density Weighted Regressioncited by 0
  19. 2025On Generalization and Distributional Update for Mimicking Observations with Adequate Explorationcited by 0
  20. 2025Robust Offline Imitation Learning Through State-level Trajectory Stitchingcited by 0
  21. 2025Minimal Ingredients for Reward Assignment from Expert Demonstrationscited by 0
  22. 2025TROFI: Trajectory-Ranked Offline Inverse Reinforcement Learningcited by 0
  23. 2025Value from Observations: Towards Large-Scale Imitation Learning via Self-Improvementcited by 0
  24. 2025Search-Based Credit Assignment for Offline Preference-Based Reinforcement Learningcited by 0
  25. 2025Bispectral OT: Dataset Comparison using Symmetry-Aware Optimal Transportcited by 0
  26. 2026Latent Wasserstein Adversarial Imitation Learningcited by 0
  27. 2026Match or Replay: Self Imitating Proximal Policy Optimizationcited by 0
  28. 2026Generalize and Guide: Decomposing Rewards for Few-Shot Inverse Reinforcement Learningcited by 0
  29. 2026Learning Multimodal One-step Flow Policy via Value-weighted Optimal Transportcited by 0
  30. 2026Differentiating Bisimulation Metrics: A Framework for Parametric Markov Chain Fitting via Bicausal Optimal Transportcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Optimal Transport Meets Reinforcement Learning: A Survey
  2. 2021Representation Matters: Offline Pretraining for Sequential Decision Making
  3. 2023Distance Weighted Supervised Learning for Offline Interaction Data
  4. 2026When Does Trajectory-Level Supervision Permit Efficient Offline Reinforcement Learning?
  5. 2023Using Offline Data to Speed Up Reinforcement Learning in Procedurally Generated Environments
  6. 2023Beyond Uniform Sampling: Offline Reinforcement Learning with Imbalanced Datasets
  7. 2021Bridging Offline Reinforcement Learning and Imitation Learning: A Tale of Pessimism
  8. 2022Semi-Supervised Offline Reinforcement Learning with Action-Free Trajectories