mlsanityData to 14 Oct 2026

Distill Knowledge in Multi-task Reinforcement Learning with Optimal-Transport Regularization

arXiv 2309.15603 ↗First posted 27 Sep 2023cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

In multi-task reinforcement learning, it is possible to improve the data efficiency of training agents by transferring knowledge from other different but related tasks. Because the experiences from different tasks are usually biased toward the specific task goals. Traditional methods rely on Kullback-Leibler regularization to stabilize the transfer of knowledge from one task to the others. In this work, we explore the direction of replacing the Kullback-Leibler divergence with a novel Optimal transport-based regularization. By using the Sinkhorn mapping, we can approximate the Optimal transport distance between the state distribution of tasks. The distance is then used as an amortized reward to regularize the amount of sharing information. We experiment our frameworks on several grid-based navigation multi-goal to validate the effectiveness of the approach. The results show that our added Optimal transport-based rewards are able to speed up the learning process of agents and outperforms several baselines on multi-task learning.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 25 of its 34 references that are in the map. First posted on arXiv 27 Sep 2023.

Builds on

25 of 34 references.

References · 25

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  3. 2015Trust Region Policy Optimizationcited by 2,580
  4. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,417
  5. 2013Sinkhorn Distances: Lightspeed Computation of Optimal Transportation Distancescited by 2,316
  6. 2020Gradient Surgery for Multi-Task Learningcited by 1,256
  7. 2017Hindsight Experience Replaycited by 1,136
  8. 2018IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architecturescited by 883
  9. 2019Behavior Regularized Offline Reinforcement Learningcited by 662
  10. 2017Wasserstein GANcited by 639
  11. 2015Policy Distillationcited by 440
  12. 2016Safe and Efficient Off-Policy Reinforcement Learningcited by 264
13 more
  1. 2017Distral: Robust Multitask Reinforcement Learningcited by 233
  2. 2021Multi-Task Reinforcement Learning with Context-based Representationscited by 174
  3. 2015Taming the Noise in Reinforcement Learning via Soft Updatescited by 127
  4. 2020Primal Wasserstein Imitation Learningcited by 121
  5. 2020COG: Connecting New Skills to Past Experience with Offline Reinforcement Learningcited by 83
  6. 2017Hierarchical and Interpretable Skill Acquisition in Multi-task Reinforcement Learningcited by 77
  7. 2022In Defense of the Unitary Scalarization for Deep Multi-Task Learningcited by 67
  8. 2021Conservative Data Sharing for Multi-Task Offline Reinforcement Learningcited by 64
  9. 2019Information asymmetry in KL-regularized RLcited by 53
  10. 2016Q(λ) with Off-Policy Correctionscited by 48
  11. 2020Imitation Learning with Sinkhorn Distancescited by 30
  12. 2021Wasserstein Unsupervised Reinforcement Learningcited by 22
  13. 2018Importance Weighted Transfer of Samples in Reinforcement Learningcited by 16

Built on by

Not cited yet.