# Temporal-Logic-Based Reward Shaping for Continuing Reinforcement Learning Tasks

arXiv: https://arxiv.org/abs/2007.01498 · HTML: https://mlsanity.com/paper/2007.01498/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 3 Jul 2020.

## Built on by (most cited first, 28 of 28)

- [Safe Reinforcement Learning via Probabilistic Logic Shields](https://mlsanity.com/paper/2303.03226/) (2023)
- [Comprehensive Overview of Reward Engineering and Shaping in Advancing Reinforcement Learning Applications](https://mlsanity.com/paper/2408.10215/) (2024)
- [Sample Efficient Model-free Reinforcement Learning from LTL Specifications with Optimality Guarantees](https://mlsanity.com/paper/2305.01381/) (2023)
- [A Survey of Continual Reinforcement Learning](https://mlsanity.com/paper/2506.21872/) (2025)
- [Reward Machines for Deep RL in Noisy and Uncertain Environments](https://mlsanity.com/paper/2406.00120/) (2024)
- [Noisy Symbolic Abstractions for Deep RL: A case study with Reward Machines](https://mlsanity.com/paper/2211.10902/) (2022)
- [Hierarchical Potential-based Reward Shaping from Task Specifications](https://mlsanity.com/paper/2110.02792/) (2021)
- [Towards Safe Autonomous Driving Policies using a Neuro-Symbolic Deep Reinforcement Learning Approach](https://mlsanity.com/paper/2307.01316/) (2023)
- [Adaptive Reward Design for Reinforcement Learning](https://mlsanity.com/paper/2412.10917/) (2024)
- [Reinforcement Learning with Knowledge Representation and Reasoning: A Brief Survey](https://mlsanity.com/paper/2304.12090/) (2023)
- [Ground-Compose-Reinforce: Grounding Language in Agentic Behaviours using Limited Data](https://mlsanity.com/paper/2507.10741/) (2025)
- [Mission-driven Exploration for Accelerated Deep Reinforcement Learning with Temporal Logic Task Specifications](https://mlsanity.com/paper/2311.17059/) (2023)
- [Informativeness of Reward Functions in Reinforcement Learning](https://mlsanity.com/paper/2402.07019/) (2024)
- [Inductive Generalization in Reinforcement Learning from Specifications](https://mlsanity.com/paper/2406.03651/) (2024)
- [Bootstrapped Reward Shaping](https://mlsanity.com/paper/2501.00989/) (2025)
- [Neuro-Symbolic Artificial Intelligence: A Task-Directed Survey in the Black-Box Models Era](https://mlsanity.com/paper/2603.03177/) (2026)
- [Learning Gait-Aware Quadruped Locomotion with Temporal Logic Specifications](https://mlsanity.com/paper/2607.00442/) (2026)
- [A Safe Preference Learning Approach for Personalization with Applications to Autonomous Vehicles](https://mlsanity.com/paper/2311.02099/) (2023)
- [Safe Policy Improvement in Constrained Markov Decision Processes](https://mlsanity.com/paper/2210.11259/) (2022)
- [Can Differentiable Decision Trees Enable Interpretable Reward Learning from Human Feedback?](https://mlsanity.com/paper/2306.13004/) (2023)
- [Sample Efficient Reinforcement Learning by Automatically Learning to Compose Subtasks](https://mlsanity.com/paper/2401.14226/) (2024)
- [Reward Augmentation in Reinforcement Learning for Testing Distributed Systems](https://mlsanity.com/paper/2409.02137/) (2024)
- [Monocular Obstacle Avoidance Based on Inverse PPO for Fixed-wing UAVs](https://mlsanity.com/paper/2411.18009/) (2024)
- [Average Reward Reinforcement Learning for Omega-Regular and Mean-Payoff Objectives](https://mlsanity.com/paper/2505.15693/) (2025)
- [Physics-Informed Reward Machines](https://mlsanity.com/paper/2508.14093/) (2025)
- [Do It for HER: First-Order Temporal Logic Reward Specification in Reinforcement Learning (Extended Version)](https://mlsanity.com/paper/2602.06227/) (2026)
- [Learning from Imperfect Demonstrations via Temporal Behavior Tree-Guided Trajectory Repair](https://mlsanity.com/paper/2604.04225/) (2026)
- [Learning-enabled Parameter Synthesis for Nonlinear Systems from Signal Temporal Logic](https://mlsanity.com/paper/2607.08899/) (2026)

## Abstract (arXiv)

In continuing tasks, average-reward reinforcement learning may be a more appropriate problem formulation than the more common discounted reward formulation. As usual, learning an optimal policy in this setting typically requires a large amount of training experiences. Reward shaping is a common approach for incorporating domain knowledge into reinforcement learning in order to speed up convergence to an optimal policy. However, to the best of our knowledge, the theoretical properties of reward shaping have thus far only been established in the discounted setting. This paper presents the first reward shaping framework for average-reward learning and proves that, under standard assumptions, the optimal policy under the original reward function can be recovered. In order to avoid the need for manual construction of the shaping function, we introduce a method for utilizing domain knowledge expressed as a temporal logic formula. The formula is automatically translated to a shaping function that provides additional reward throughout the learning process. We evaluate the proposed method on three continuing tasks. In all cases, shaping speeds up the average-reward learning rate without any reduction in the performance of the learned policy compared to relevant baselines.