mlsanityData to 7 Oct 2026

Episodic Policy Gradient Training

arXiv 2112.01853 ↗First posted 3 Dec 2021cs.LG cs.MAcited by 5

Not on the map. Why?

Abstract (arXiv)

We introduce a novel training procedure for policy gradient methods wherein episodic memory is used to optimize the hyperparameters of reinforcement learning algorithms on-the-fly. Unlike other hyperparameter searches, we formulate hyperparameter scheduling as a standard Markov Decision Process and use episodic memory to store the outcome of used hyperparameters and their training contexts. At any policy update step, the policy learner refers to the stored experiences, and adaptively reconfigures its learning algorithm with the new hyperparameters determined by the memory. This mechanism, dubbed as Episodic Policy Gradient Training (EPGT), enables an episodic learning process, and jointly learns the policy and the learning algorithm's hyperparameters within a single run. Experimental results on both continuous and discrete environments demonstrate the advantage of using the proposed method in boosting the performance of various policy gradient algorithms.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Builds on 24 of its 43 references that are in the map. First posted on arXiv 3 Dec 2021.

Builds on

24 of 43 references.

References · 24

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  3. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,218
  4. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  5. 2017Deep Reinforcement Learning that Matterscited by 727
  6. 2016Benchmarking Deep Reinforcement Learning for Continuous Controlcited by 361
  7. 2017Population Based Training of Neural Networkscited by 351
  8. 2018Meta-Gradient Reinforcement Learningcited by 156
  9. 2017Neural Episodic Controlcited by 148
  10. 2016Fast Bayesian Optimization of Machine Learning Hyperparameters on Large Datasetscited by 131
  11. 2017Scalable trust-region method for deep reinforcement learning using Kronecker-factored approximationcited by 119
  12. 2015Batch Bayesian Optimization via Local Penalizationcited by 103
12 more
  1. 2016Model-Free Episodic Controlcited by 100
  2. 2015Scalable Gradient-Based Tuning of Continuous Regularization Hyperparameterscited by 71
  3. 2020Provably Efficient Online Hyperparameter Optimization with Population-Based Banditscited by 61
  4. 2021On the Importance of Hyperparameter Optimization for Model-based Reinforcement Learningcited by 59
  5. 2018Optimizing Agent Behavior over Long Time Scales by Transporting Valuecited by 57
  6. 2018Bayesian Optimization in AlphaGocited by 55
  7. 2020Self-Attentive Associative Memorycited by 33
  8. 2019A Generalized Framework for Population Based Trainingcited by 22
  9. 2019Learning to Remember More with Less Memorizationcited by 14
  10. 2021Model-Based Episodic Memory Induces Dynamic Hybrid Controlscited by 14
  11. 2020Online Hyper-parameter Tuning in Off-policy Learning via Evolutionary Strategiescited by 9
  12. 2020Neurocoder: Learning General-Purpose Computation Using Stored Neural Programscited by 6

Built on by

Cited by year

Citing papers per year: 2022: 1; 2023: 1; 2024: 2; 2025: 1.

5 papers.

  1. 2024Stable Hadamard Memory: Revitalizing Memory-Augmented Agents for Reinforcement Learningcited by 8
  2. 2023Neural Episodic Control with State Abstractioncited by 7
  3. 2025Reasoning Under 1 Billion: Memory-Augmented Reinforcement Learning for Large Language Modelscited by 6
  4. 2024Large Language Models Prompting With Episodic Memorycited by 4
  5. 2022Improving Document Image Understanding with Reinforcement Finetuningcited by 0