Sample-Efficient Reinforcement Learning with Temporal Logic Objectives: Leveraging the Task Specification to Guide Exploration
Not on the map. Why?
Abstract (arXiv)
This paper addresses the problem of learning optimal control policies for systems with uncertain dynamics and high-level control objectives specified as Linear Temporal Logic (LTL) formulas. Uncertainty is considered in the workspace structure and the outcomes of control decisions giving rise to an unknown Markov Decision Process (MDP). Existing reinforcement learning (RL) algorithms for LTL tasks typically rely on exploring a product MDP state-space uniformly (using e.g., an ε-greedy policy) compromising sample-efficiency. This issue becomes more pronounced as the rewards get sparser and the MDP size or the task complexity increase. In this paper, we propose an accelerated RL algorithm that can learn control policies significantly faster than competitive approaches. Its sample-efficiency relies on a novel task-driven exploration strategy that biases exploration towards directions that may contribute to task satisfaction. We provide theoretical analysis and extensive comparative experiments demonstrating the sample-efficiency of the proposed method. The benefit of our method becomes more evident as the task complexity or the MDP size increases.
Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 26 of its 55 references that are in the map. First posted on arXiv 16 Oct 2024.
Builds on
26 of 55 references.
References · 26
- 2017Curiosity-driven Exploration by Self-supervised Predictioncited by 1,173
- 2020Deep Reinforcement Learning for Autonomous Driving: A Surveycited by 698
- 2020Reward Machines: Exploiting Reward Function Structure in Reinforcement Learningcited by 183
- 2019Reinforcement Learning for Temporal Logic Control Synthesis with Probabilistic Satisfaction Guaranteescited by 77
- 2018Omega-Regular Objectives in Model-Free Reinforcement Learningcited by 74
- 2019Control Synthesis from Linear Temporal Logic Specifications using Model-Free Reinforcement Learningcited by 74
- 2017Boltzmann Exploration Done Rightcited by 69
- 2021A Survey of Exploration Methods in Reinforcement Learningcited by 68
- 2021Compositional Reinforcement Learning from Logical Specificationscited by 65
- 2014Probably Approximately Correct MDP Learning and Control With Temporal Logic Constraintscited by 53
- 2014Verification of Markov Decision Processes using Learning Algorithmscited by 38
- 2019Certified Reinforcement Learning with Logic Guidancecited by 35
14 more
- 2019Reinforcement Learning with Probabilistic Guarantees for Autonomous Drivingcited by 27
- 2018STyLuS*: A Temporal Logic Optimal Control Synthesis Algorithm for Large-Scale Multi-Robot Systemscited by 24
- 2021Computational Benefits of Intermediate Rewards for Goal-Reaching Policy Learningcited by 17
- 2020Formal Controller Synthesis for Continuous-Space MDPs via Model-Free Reinforcement Learningcited by 15
- 2020Optimal Probabilistic Motion Planning with Potential Infeasible LTL Constraintscited by 14
- 2020Perception-Based Temporal Logic Planning in Uncertain Semantic Mapscited by 12
- 2022Overcoming Exploration: Deep Reinforcement Learning for Continuous Control in Cluttered Environments from Temporal Logic Specificationscited by 11
- 2022LCRL: Certified Policy Synthesis via Logically-Constrained Reinforcement Learningcited by 11
- 2017Probabilistic Motion Planning under Temporal Tasks and Soft Constraintscited by 9
- 2022Accelerated Reinforcement Learning for Temporal Logic Control Objectivescited by 9
- 2022Model-Free Reinforcement Learning for Symbolic Automata-encoded Objectivescited by 6
- 2022Learning Minimally-Violating Continuous Control for Infeasible Linear Temporal Logic Specificationscited by 5
- 2021Learning Optimal Strategies for Temporal Tasks in Stochastic Gamescited by 3
- 2024On the Uniqueness of Solution for the Bellman Equation of LTL Objectivescited by 3
Built on by
Citing papers per year: 2025: 1; 2026: 2.
3 papers.