mlsanityData to 7 Oct 2026

Leveraging Optimal Transport for Enhanced Offline Reinforcement Learning in Surgical Robotic Environments

arXiv 2310.08841 ↗First posted 13 Oct 2023cs.AI cs.RO stat.MLcited by 3

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

Most Reinforcement Learning (RL) methods are traditionally studied in an active learning setting, where agents directly interact with their environments, observe action outcomes, and learn through trial and error. However, allowing partially trained agents to interact with real physical systems poses significant challenges, including high costs, safety risks, and the need for constant supervision. Offline RL addresses these cost and safety concerns by leveraging existing datasets and reducing the need for resource-intensive real-time interactions. Nevertheless, a substantial challenge lies in the demand for these datasets to be meticulously annotated with rewards. In this paper, we introduce Optimal Transport Reward (OTR) labelling, an innovative algorithm designed to assign rewards to offline trajectories, using a small number of high-quality expert demonstrations. The core principle of OTR involves employing Optimal Transport (OT) to calculate an optimal alignment between an unlabeled trajectory from the dataset and an expert demonstration. This alignment yields a similarity measure that is effectively interpreted as a reward signal. An offline RL algorithm can then utilize these reward signals to learn a policy. This approach circumvents the need for handcrafted rewards, unlocking the potential to harness vast datasets for policy learning. Leveraging the SurRoL simulation platform tailored for surgical robot learning, we generate datasets and employ them to train policies using the OTR algorithm. By demonstrating the efficacy of OTR in a different domain, we emphasize its versatility and its potential to expedite RL deployment across a wide range of fields.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 23 of its 30 references that are in the map. First posted on arXiv 13 Oct 2023.

Builds on

23 of 30 references.

Closest ancestors · 6

Same direction, newest first

  1. 2023Optimal Transport for Offline Imitation Learning24 Mar
  2. 2022How to Leverage Unlabeled Data in Offline Reinforcement Learning3 Feb
  3. 2021Offline Reinforcement Learning with Implicit Q-Learning12 Oct
  4. 2020Critic Regularized Regression26 Jun
  5. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
  6. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May

Other references · 17

  1. 2015Continuous control with deep reinforcement learningcited by 3,279
  2. 2013Sinkhorn Distances: Lightspeed Computation of Optimal Transportation Distancescited by 2,277
  3. 2018Soft Actor-Critic Algorithms and Applicationscited by 1,179
  4. 2018An Algorithmic Perspective on Imitation Learningcited by 455
  5. 2017Soft-DTW: a Differentiable Loss Function for Time-Seriescited by 265
  6. 2023A Survey of Imitation Learning: Algorithms, Recent Developments, and Challengescited by 145
  7. 2020Primal Wasserstein Imitation Learningcited by 120
  8. 2019End-to-End Robotic Reinforcement Learning without Reward Engineeringcited by 115
  9. 2022Watch and Match: Supercharging Imitation with Regularized Optimal Transportcited by 64
  10. 2021Cross-Domain Imitation Learning via Optimal Transportcited by 49
  11. 2020CO-Optimal Transportcited by 42
  12. 2020Imitation Learning with Sinkhorn Distancescited by 30
5 more
  1. 2021SurRoL: An Open-source Reinforcement Learning Centered and dVRK Compatible Platform for Surgical Robot Learningcited by 28
  2. 2023Demonstration-Guided Reinforcement Learning with Efficient Exploration for Task Automation of Surgical Robotcited by 21
  3. 2022Real World Offline Reinforcement Learning with Realistic Data Sourcecited by 16
  4. 2020Aligning Time Series on Incomparable Spacescited by 15
  5. 2022Goal-Aware Generative Adversarial Imitation Learning from Imperfect Demonstration for Robotic Cloth Manipulationcited by 7

Built on by

Cited by year

Citing papers per year: 2024: 2; 2025: 1.

3 papers.

  1. 2024Optimal Transport-Assisted Risk-Sensitive Q-Learningcited by 3
  2. 2025Latent Embedding Adaptation for Human Preference Alignment in Diffusion Plannerscited by 1
  3. 2024Off-dynamics Conditional Diffusion Plannerscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Optimal Transport Meets Reinforcement Learning: A Survey
  2. 2020Semi-supervised reward learning for offline reinforcement learning
  3. 2024Real-World Offline Reinforcement Learning from Vision Language Model Feedback
  4. 2024Rethinking Optimal Transport in Offline Reinforcement Learning
  5. 2024Align Your Intents: Offline Imitation Learning via Optimal Transport
  6. 2026When Does Trajectory-Level Supervision Permit Efficient Offline Reinforcement Learning?
  7. 2021A Workflow for Offline Model-Free Robotic Reinforcement Learning
  8. 2025DARIL: When Imitation Learning outperforms Reinforcement Learning in Surgical Action Planning