Data to 7 Oct 2026

Heuristic-Guided Reinforcement Learning

arXiv 2106.02757 ↗First posted 5 Jun 2021cs.LG cs.AIcited by 41

Not on the map. Why?

Abstract (arXiv)

We provide a framework for accelerating reinforcement learning (RL) algorithms by heuristics constructed from domain knowledge or offline data. Tabula rasa RL algorithms require environment interactions or computation that scales with the horizon of the sequential decision-making task. Using our framework, we show how heuristic-guided RL induces a much shorter-horizon subproblem that provably solves the original task. Our framework can be viewed as a horizon-based regularization for controlling bias and variance in RL under a finite interaction budget. On the theoretical side, we characterize properties of a good heuristic and its impact on RL acceleration. In particular, we introduce the novel concept of an improvable heuristic, a heuristic that allows an RL agent to extrapolate beyond its prior knowledge. On the empirical side, we instantiate our framework to accelerate several state-of-the-art algorithms in simulated robotic control tasks and procedurally generated games. Our framework complements the rich literature on warm-starting RL with expert demonstrations or exploratory datasets, and introduces a principled method for injecting prior knowledge into RL.

Key facts. Cited by 41 papers in the map (data through 7 Oct 2026). Builds on 30 of its 72 references that are in the map. First posted on arXiv 5 Jun 2021.

Builds on

30 of 72 references.

References · 30

2017Proximal Policy Optimization Algorithmscited by 14,441
2015Distilling the Knowledge in a Neural Networkcited by 10,076
2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 1,916
2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,777
2019Dota 2 with Large Scale Deep Reinforcement Learningcited by 1,084
2018IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architecturescited by 866
2020AWAC: Accelerating Online Reinforcement Learning with Offline Datasetscited by 647
2018Is Q-learning Provably Efficient?cited by 485
2019Leveraging Procedural Generation to Benchmark Reinforcement Learningcited by 478
2020Is Pessimism Provably Efficient for Offline RL?cited by 362
2020Beyond UCB: Optimal and Efficient Contextual Bandits with Regression Oraclescited by 195

18 more

2017Reverse Curriculum Generation for Reinforcement Learningcited by 185
2020Learning to Utilize Shaping Rewards: A New Approach of Reward Shapingcited by 112
2015Sample Complexity of Episodic Fixed-Horizon Reinforcement Learningcited by 105
2017Deeply AggreVaTeD: Differentiable Imitation Learning for Sequential Predictioncited by 93
2014The LAMA Planner: Guiding Cost-Based Anytime Planning with Landmarkscited by 86
2020Discount Factor as a Regularizer in Reinforcement Learningcited by 44
2018Truncated Horizon Policy Search: Combining Reinforcement Learning & Imitation Learningcited by 39
2021Regularized Behavior Value Estimationcited by 32
2020Blending MPC & Value Function Approximation for Efficient Reinforcement Learningcited by 27
2020Understanding the Power and Limitations of Teaching with Imperfect Knowledgecited by 23
2021Measuring Sample Efficiency and Generalization in Reinforcement Learning Benchmarks: NeurIPS 2020 Procgen Benchmarkcited by 23
2018Beyond the One Step Greedy Approach in Reinforcement Learningcited by 20
2019Using a Logarithmic Mapping to Enable Lower Discount Factors in Reinforcement Learningcited by 16
2019Separating value functions across time-scalescited by 15
2019Deep Value Model Predictive Controlcited by 14
2019Gamma-Nets: Generalizing Value Estimation over Timescalecited by 6
2018Planning with a Receding Horizon for Manipulation in Clutter using a Learned Value Functioncited by 5
2020Reward Tweaking: Maximizing the Total Reward While Planning for Short Horizonscited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 3; 2022: 7; 2023: 9; 2024: 13; 2025: 6; 2026: 3.

41 papers.

Sort:

By area:

2022Adversarially Trained Actor Critic for Offline Reinforcement Learningcited by 111
2022Reincarnating Reinforcement Learning: Reusing Prior Computation to Accelerate Progresscited by 63
2022Unpacking Reward Shaping: Understanding the Benefits of Reward Engineering on Sample Complexitycited by 59
2021Safe Reinforcement Learning Using Advantage-Based Interventioncited by 45
2021Deep Multiagent Reinforcement Learning: Challenges and Directionscited by 42
2022Hindsight Learning for MDPs with Exogenous Inputscited by 25
2024Highly Efficient Self-Adaptive Reward Shaping for Reinforcement Learningcited by 21
2024Doubly Mild Generalization for Offline Reinforcement Learningcited by 21
2025MONA: Myopic Optimization with Non-myopic Approval Can Mitigate Multi-step Reward Hackingcited by 19
2021Improving Zero-shot Generalization in Offline Reinforcement Learning using Generalized Similarity Functionscited by 17
2023TGRL: An Algorithm for Teacher Guided Reinforcement Learningcited by 16
2025Rapidly Adapting Policies to the Real World via Simulation-Guided Fine-Tuningcited by 15

29 more

2022Data-Efficient Pipeline for Offline Reinforcement Learning with Limited Datacited by 12
2023Improving Offline RL by Blending Heuristicscited by 10
2024On the Modeling Capabilities of Large Language Models for Sequential Decision Makingcited by 9
2026Understanding the Challenges in Iterative Generative Optimization with LLMscited by 8
2023Accelerating exploration and representation learning with offline pre-trainingcited by 7
2023Towards an On-device Agent for Text Rewritingcited by 7
2024Learning Logic Specifications for Policy Guidance in POMDPs: an Inductive Logic Programming Approachcited by 7
2024Extracting Heuristics from Large Language Models for Reward Shaping in Reinforcement Learningcited by 7
2024The Dark Side of Rich Rewards: Understanding and Mitigating Noise in VLM Rewardscited by 7
2022Making Reinforcement Learning Work on Swimmercited by 6
2023SACHA: Soft Actor-Critic with Heuristic-Based Attention for Partially Observable Multi-Agent Path Findingcited by 6
2023On Using Admissible Bounds for Learning Forward Search Heuristicscited by 5
2022Reinforcement learning with Demonstrations from Mismatched Task under Sparse Rewardcited by 4
2024Proofread: Fixes All Errors with One Tapcited by 4
2025Online inductive learning from answer sets for efficient reinforcement learning explorationcited by 4
2023Provable Reset-free Reinforcement Learning by No-Regret Reductioncited by 3
2023Toward Computationally Efficient Inverse Reinforcement Learning via Reward Shapingcited by 2
2024Enhancing Q-Learning with Large Language Model Heuristicscited by 2
2025Integrating Human Knowledge Through Action Masking in Reinforcement Learning for Operations Researchcited by 2
2024On the Sample Efficiency of Abstractions and Potential-Based Reward Shaping in Reinforcement Learningcited by 1
2024Dense Dynamics-Aware Reward Synthesis: Integrating Prior Experience with Demonstrationscited by 1
2024Fairness in Reinforcement Learning with Bisimulation Metricscited by 1
2023Adaptive Services Function Chain Orchestration For Digital Health Twin Use Cases: Heuristic-boosted Q-Learning Approachcited by 0
2024Heuristic Algorithm-based Action Masking Reinforcement Learning (HAAM-RL) with Ensemble Inference Methodcited by 0
2024Emotion-Agent: Unsupervised Deep Reinforcement Learning with Distribution-Prototype Reward for Continuous Emotional EEG Analysiscited by 0
2025Optimizing 2D+1 Packing in Constrained Environments Using Deep Reinforcement Learningcited by 0
2025Learning Upper Lower Value Envelopes to Shape Online RL: A Principled Approachcited by 0
2026Sample-efficient Neuro-symbolic Proximal Policy Optimizationcited by 0
2026Hierarchical Reinforcement Learning for Sparse-Reward Search in Commutative Algebracited by 0