Mission-driven Exploration for Accelerated Deep Reinforcement Learning with Temporal Logic Task Specifications
Not on the map. Why?
Abstract (arXiv)
This paper addresses the problem of designing control policies for agents with unknown stochastic dynamics and control objectives specified using Linear Temporal Logic (LTL). Recent Deep Reinforcement Learning (DRL) algorithms have aimed to compute policies that maximize the satisfaction probability of LTL formulas, but they often suffer from slow learning performance. To address this, we introduce a novel Deep Q-learning algorithm that significantly improves learning speed. The enhanced sample efficiency stems from a mission-driven exploration strategy that prioritizes exploration towards directions likely to contribute to mission success. Identifying these directions relies on an automaton representation of the LTL task as well as a learned neural network that partially models the agent-environment interaction. We provide comparative experiments demonstrating the efficiency of our algorithm on robot navigation tasks in unseen environments.
Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 37 of its 54 references that are in the map. First posted on arXiv 28 Nov 2023.
Builds on
37 of 54 references.
References · 37
- 2017Proximal Policy Optimization Algorithmscited by 14,441
- 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
- 2013Playing Atari with Deep Reinforcement Learningcited by 3,211
- 2015Deep Reinforcement Learning with Double Q-learningcited by 1,704
- 2017Curiosity-driven Exploration by Self-supervised Predictioncited by 1,173
- 2017Rainbow: Combining Improvements in Deep Reinforcement Learningcited by 774
- 2020Deep Reinforcement Learning for Autonomous Driving: A Surveycited by 698
- 2020Reward Machines: Exploiting Reward Function Structure in Reinforcement Learningcited by 183
- 2017Socially Aware Motion Planning with Deep Reinforcement Learningcited by 128
- 2019Reinforcement Learning for Temporal Logic Control Synthesis with Probabilistic Satisfaction Guaranteescited by 77
- 2018Omega-Regular Objectives in Model-Free Reinforcement Learningcited by 74
- 2019Control Synthesis from Linear Temporal Logic Specifications using Model-Free Reinforcement Learningcited by 74
25 more
- 2021Compositional Reinforcement Learning from Logical Specificationscited by 65
- 2017PRM-RL: Long-range Robotic Navigation Tasks by Combining Reinforcement Learning and Sampling-based Planningcited by 64
- 2014Probably Approximately Correct MDP Learning and Control With Temporal Logic Constraintscited by 53
- 2014Verification of Markov Decision Processes using Learning Algorithmscited by 38
- 2021Modular Deep Reinforcement Learning for Continuous Motion Planning with Temporal Logiccited by 38
- 2019Transfer of Temporal Logic Formulas in Reinforcement Learningcited by 31
- 2020Temporal-Logic-Based Reward Shaping for Continuing Reinforcement Learning Taskscited by 28
- 2018STyLuS*: A Temporal Logic Optimal Control Synthesis Algorithm for Large-Scale Multi-Robot Systemscited by 24
- 2020Reinforcement Learning Based Temporal Logic Control with Maximum Probabilistic Satisfactioncited by 22
- 2021Computational Benefits of Intermediate Rewards for Goal-Reaching Policy Learningcited by 17
- 2023Sample Efficient Model-free Reinforcement Learning from LTL Specifications with Optimality Guaranteescited by 17
- 2020Formal Controller Synthesis for Continuous-Space MDPs via Model-Free Reinforcement Learningcited by 15
- 2020Perception-Based Temporal Logic Planning in Uncertain Semantic Mapscited by 12
- 2023A PAC Learning Algorithm for LTL and Omega-regular Objectives in MDPscited by 12
- 2022Overcoming Exploration: Deep Reinforcement Learning for Continuous Control in Cluttered Environments from Temporal Logic Specificationscited by 11
- 2022LCRL: Certified Policy Synthesis via Logically-Constrained Reinforcement Learningcited by 11
- 2021Model-Free Reinforcement Learning for Optimal Control of MarkovDecision Processes Under Signal Temporal Logic Specificationscited by 10
- 2022Exploiting Transformer in Sparse Reward Reinforcement Learning for Interpretable Temporal Logic Motion Planningcited by 10
- 2017Probabilistic Motion Planning under Temporal Tasks and Soft Constraintscited by 9
- 2021Programmatic Reward Design by Examplecited by 9
- 2022Accelerated Reinforcement Learning for Temporal Logic Control Objectivescited by 9
- 2019DeepSynth: Automata Synthesis for Automatic Task Segmentation in Deep Reinforcement Learningcited by 8
- 2020Continuous Motion Planning with Temporal Logic Specifications using Deep Neural Networkscited by 8
- 2023Identify, Estimate and Bound the Uncertainty of Reinforcement Learning for Autonomous Drivingcited by 6
- 2022Learning Minimally-Violating Continuous Control for Infeasible Linear Temporal Logic Specificationscited by 5
Built on by
Citing papers per year: 2024: 2; 2025: 1.
3 papers.