mlsanityData to 7 Oct 2026

Eventual Discounting Temporal Logic Counterfactual Experience Replay

arXiv 2303.02135 ↗First posted 3 Mar 2023cs.LG cs.AI cs.LOcited by 18

Not on the map. Why?

Abstract (arXiv)

Linear temporal logic (LTL) offers a simplified way of specifying tasks for policy optimization that may otherwise be difficult to describe with scalar reward functions. However, the standard RL framework can be too myopic to find maximally LTL satisfying policies. This paper makes two contributions. First, we develop a new value-function based proxy, using a technique we call eventual discounting, under which one can find policies that satisfy the LTL specification with highest achievable probability. Second, we develop a new experience replay method for generating off-policy data from on-policy rollouts via counterfactual reasoning on different ways of satisfying the LTL specification. Our experiments, conducted in both discrete and continuous state-action spaces, confirm the effectiveness of our counterfactual experience replay approach.

Key facts. Cited by 18 papers in the map (data through 7 Oct 2026). Builds on 18 of its 35 references that are in the map. First posted on arXiv 3 Mar 2023.

Builds on

18 of 35 references.

References · 18

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2013Playing Atari with Deep Reinforcement Learningcited by 3,211
  3. 2015Deep Reinforcement Learning with Double Q-learningcited by 1,704
  4. 2018Reward learning from human preferences and demonstrations in Ataricited by 314
  5. 2020Reward Machines: Exploiting Reward Function Structure in Reinforcement Learningcited by 183
  6. 2019Control Synthesis from Linear Temporal Logic Specifications using Model-Free Reinforcement Learningcited by 74
  7. 2021LTL2Action: Generalizing LTL Instructions for Multi-Task RLcited by 72
  8. 2021On the Expressivity of Markov Rewardcited by 71
  9. 2021On the Importance of Hyperparameter Optimization for Model-based Reinforcement Learningcited by 59
  10. 2014Probably Approximately Correct MDP Learning and Control With Temporal Logic Constraintscited by 53
  11. 2018Logically-Constrained Reinforcement Learningcited by 50
  12. 2014A Learning Based Approach to Control Synthesis of Markov Decision Processes for Linear Temporal Logic Specificationscited by 49
6 more
  1. 2021Modular Deep Reinforcement Learning for Continuous Motion Planning with Temporal Logiccited by 38
  2. 2022Policy Optimization with Linear Temporal Logic Constraintscited by 19
  3. 2021On the (In)Tractability of Reinforcement Learning for LTL Objectivescited by 18
  4. 2019Is Deep Reinforcement Learning Really Superhuman on Atari? Leveling the playing fieldcited by 14
  5. 2020Optimal Probabilistic Motion Planning with Potential Infeasible LTL Constraintscited by 14
  6. 2020Continuous Motion Planning with Temporal Logic Specifications using Deep Neural Networkscited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 2; 2024: 6; 2025: 4; 2026: 6.

18 papers.

Sort:

By area:

  1. 2024DeepLTL: Learning to Efficiently Satisfy Complex LTL Specifications for Multi-Task RLcited by 20
  2. 2024Reinforcement Learning with LTL and ω-Regular Objectives via Optimality-Preserving Translation to Average Rewardscited by 13
  3. 2024LTL-Constrained Policy Optimization with Cycle Experience Replaycited by 10
  4. 2025One Subgoal at a Time: Zero-Shot Generalization to Arbitrary Linear Temporal Logic Requirements in Multi-Task Reinforcement Learningcited by 8
  5. 2024Directed Exploration in Reinforcement Learning from Linear Temporal Logiccited by 7
  6. 2023Reinforcement Learning with Knowledge Representation and Reasoning: A Brief Surveycited by 5
  7. 2025Ground-Compose-Reinforce: Grounding Language in Agentic Behaviours using Limited Datacited by 5
  8. 2023A Counterfactual Safety Margin Perspective on the Scoring of Autonomous Vehicles' Riskinesscited by 3
  9. 2024On the Uniqueness of Solution for the Bellman Equation of LTL Objectivescited by 3
  10. 2026Formal Methods Meet LLMs: Auditing, Monitoring, and Intervention for Compliance of Advanced AI Systemscited by 3
  11. 2026PlatoLTL: Scaling LTL-Guided Multi-Task RLcited by 2
  12. 2026Semantically Labelled Automata for Multi-Task Reinforcement Learning with LTL Instructionscited by 2
6 more
  1. 2024Generalization of Compositional Tasks with Logical Specification via Implicit Planningcited by 1
  2. 2025Accelerated Learning with Linear Temporal Logic using Differentiable Simulationcited by 1
  3. 2026Zero-Shot Instruction Following in RL via Structured LTL Representationscited by 1
  4. 2026SpecRLBench: A Benchmark for Generalization in Specification-Guided Reinforcement Learningcited by 1
  5. 2025Zero-Shot Instruction Following in RL via Structured LTL Representationscited by 0
  6. 2026Jaxolotl: A Unified High-Performance Benchmark Suite for LTL-Based Multi-Task RLcited by 0