mlsanityData to 7 Oct 2026

Robot Policy Learning with Temporal Optimal Transport Reward

arXiv 2410.21795 ↗First posted 29 Oct 2024cs.AI cs.LG cs.ROcited by 8

Not on the map. Why?

Abstract (arXiv)

Reward specification is one of the most tricky problems in Reinforcement Learning, which usually requires tedious hand engineering in practice. One promising approach to tackle this challenge is to adopt existing expert video demonstrations for policy learning. Some recent work investigates how to learn robot policies from only a single/few expert video demonstrations. For example, reward labeling via Optimal Transport (OT) has been shown to be an effective strategy to generate a proxy reward by measuring the alignment between the robot trajectory and the expert demonstrations. However, previous work mostly overlooks that the OT reward is invariant to temporal order information, which could bring extra noise to the reward signal. To address this issue, in this paper, we introduce the Temporal Optimal Transport (TemporalOT) reward to incorporate temporal order information for learning a more accurate OT-based proxy reward. Extensive experiments on the Meta-world benchmark tasks validate the efficacy of the proposed method. Code is available at: https://github.com/fuyw/TemporalOT

Key facts. Cited by 8 papers in the map (data through 7 Oct 2026). Builds on 38 of its 61 references that are in the map. First posted on arXiv 29 Oct 2024.

Builds on

38 of 61 references.

References · 38

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  4. 2022Training language models to follow instructions with human feedbackcited by 15,712
  5. 2013Sinkhorn Distances: Lightspeed Computation of Optimal Transportation Distancescited by 2,277
  6. 2016Generative Adversarial Imitation Learningcited by 1,576
  7. 2019Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learningcited by 1,168
  8. 2018Exploration by Random Network Distillationcited by 966
  9. 2022R3M: A Universal Visual Representation for Robot Manipulationcited by 678
  10. 2020AWAC: Accelerating Online Reinforcement Learning with Offline Datasetscited by 647
  11. 2015Deep Recurrent Q-Learning for Partially Observable MDPscited by 457
  12. 2021Implicit Behavioral Cloningcited by 442
26 more
  1. 2022VIP: Towards Universal Visual Reward and Representation via Value-Implicit Pre-Trainingcited by 384
  2. 2021Mastering Visual Continuous Control: Improved Data-Augmented Reinforcement Learningcited by 359
  3. 2017Overcoming Exploration in Reinforcement Learning with Demonstrationscited by 322
  4. 2022Transformers are Sample-Efficient World Modelscited by 317
  5. 2016Cooperative Inverse Reinforcement Learningcited by 298
  6. 2018Generative Adversarial Imitation from Observationcited by 183
  7. 2020Reward Machines: Exploiting Reward Function Structure in Reinforcement Learningcited by 183
  8. 2023LIV: Language-Image Representations and Rewards for Robotic Controlcited by 180
  9. 2023RoboCLIP: One Demonstration is Enough to Learn Robot Policiescited by 128
  10. 2019End-to-End Robotic Reinforcement Learning without Reward Engineeringcited by 115
  11. 2021State Entropy Maximization with Random Encoders for Efficient Explorationcited by 107
  12. 2023Policy Expansion for Bridging Offline-to-Online Reinforcement Learningcited by 103
  13. 2022A Walk in the Park: Learning to Walk in 20 Minutes With Model-Free Reinforcement Learningcited by 76
  14. 2021Demonstration-Guided Reinforcement Learning with Learned Skillscited by 73
  15. 2022Watch and Match: Supercharging Imitation with Regularized Optimal Transportcited by 64
  16. 2023Optimal Transport for Offline Imitation Learningcited by 42
  17. 2021A Survey on Optimal Transport for Machine Learning: Theory and Applicationscited by 35
  18. 2023Fast Rates for Maximum Entropy Explorationcited by 32
  19. 2022Human-level Atari 200x fastercited by 28
  20. 2024FuRL: Visual-Language Models as Fuzzy Rewards for Reinforcement Learningcited by 26
  21. 2021Wasserstein Unsupervised Reinforcement Learningcited by 21
  22. 2023MAHALO: Unifying Offline Reinforcement Learning and Imitation Learning from Observationscited by 18
  23. 2019Unnormalized Optimal Transportcited by 15
  24. 2023Learning from Visual Observation via Offline Pretrained State-to-Go Transformercited by 14
  25. 2023CEIL: Generalized Contextual Imitation Learningcited by 12
  26. 2023Imitation Learning from Observation with Automatic Discount Schedulingcited by 10

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 4; 2026: 4.

8 papers.

  1. 2026Robometer: Scaling General-Purpose Robotic Reward Models via Trajectory Comparisonscited by 51
  2. 2025VLP: Vision-Language Preference Learning for Embodied Manipulationcited by 8
  3. 2025Imitation Learning from a Single Temporally Misaligned Videocited by 7
  4. 2025TimeRewarder: Learning Dense Reward from Passive Videos via Frame-wise Temporal Distancecited by 5
  5. 2026RARM: Confidence-Gated Progress Reward Modeling for RL in Manipulationcited by 3
  6. 2026ReGIL: Retrieval-Guided Imitation Learning from a Single Demonstrationcited by 1
  7. 2026UR-VC: Unsupervised Robotic Value Correction for Time-Derived Progress Proxiescited by 1
  8. 2025Minimal Ingredients for Reward Assignment from Expert Demonstrationscited by 0