mlsanityData to 14 Oct 2026

Reward-Machine-Guided, Self-Paced Reinforcement Learning

arXiv 2305.16505 ↗First posted 25 May 2023cs.LGcited by 1

Not on the map. Why?

Abstract (arXiv)

Self-paced reinforcement learning (RL) aims to improve the data efficiency of learning by automatically creating sequences, namely curricula, of probability distributions over contexts. However, existing techniques for self-paced RL fail in long-horizon planning tasks that involve temporally extended behaviors. We hypothesize that taking advantage of prior knowledge about the underlying task structure can improve the effectiveness of self-paced RL. We develop a self-paced RL algorithm guided by reward machines, i.e., a type of finite-state machine that encodes the underlying task structure. The algorithm integrates reward machines in 1) the update of the policy and value functions obtained by any RL algorithm of choice, and 2) the update of the automated curriculum that generates context distributions. Our empirical results evidence that the proposed algorithm achieves optimal behavior reliably even in cases in which existing baselines cannot make any meaningful progress. It also decreases the curriculum length and reduces the variance in the curriculum generation process by up to one-fourth and four orders of magnitude, respectively.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 22 of its 43 references that are in the map. First posted on arXiv 25 May 2023.

Builds on

22 of 43 references.

References · 22

  1. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  2. 2017Hindsight Experience Replaycited by 1,136
  3. 2020Curriculum Learning for Reinforcement Learning Domains: A Framework and Surveycited by 373
  4. 2017Automatic Goal Generation for Reinforcement Learning Agentscited by 259
  5. 2015Contextual Markov Decision Processescited by 220
  6. 2017Reverse Curriculum Generation for Reinforcement Learningcited by 198
  7. 2020Prioritized Level Replaycited by 198
  8. 2019Teacher algorithms for curriculum learning of Deep RL in continuously parameterized environmentscited by 119
  9. 2016Reinforcement Learning With Temporal Logic Rewardscited by 117
  10. 2021Replay-Guided Adversarial Environment Designcited by 111
  11. 2020Automatic Curriculum Learning through Value Disagreementcited by 87
  12. 2017Environment-Independent Task Specifications via GLTLcited by 67
10 more
  1. 2019Joint Inference of Reward Machines and Policies for Reinforcement Learningcited by 56
  2. 2020Self-Paced Deep Reinforcement Learningcited by 50
  3. 2020Encoding formulas as deep networks: Reinforcement learning for zero-shot execution of LTL formulascited by 36
  4. 2019Transfer of Temporal Logic Formulas in Reinforcement Learningcited by 31
  5. 2019Automated curricula through setter-solver interactionscited by 26
  6. 2019Self-Paced Contextual Reinforcement Learningcited by 26
  7. 2021Variational Automatic Curriculum Learning for Sparse-Reward Cooperative Multi-Agent Problemscited by 26
  8. 2021Self-Paced Context Evaluation for Contextual Reinforcement Learningcited by 25
  9. 2021A Probabilistic Interpretation of Self-Paced Learning with Applications to Reinforcement Learningcited by 22
  10. 2021Lifelong Reinforcement Learning with Temporal Logic Formulas and Reward Machinescited by 5

Built on by

Cited by year

Citing papers per year: 2023: 1.

1 paper.

  1. 2023Domain Randomization via Entropy Maximizationcited by 19