mlsanityData to 7 Oct 2026

Compositional Automata Embeddings for Goal-Conditioned Reinforcement Learning

arXiv 2411.00205 ↗First posted 31 Oct 2024cs.LG cs.AI cs.CLcited by 18

Not on the map. Why?

Abstract (arXiv)

Goal-conditioned reinforcement learning is a powerful way to control an AI agent's behavior at runtime. That said, popular goal representations, e.g., target states or natural language, are either limited to Markovian tasks or rely on ambiguous task semantics. We propose representing temporal goals using compositions of deterministic finite automata (cDFAs) and use cDFAs to guide RL agents. cDFAs balance the need for formal temporal semantics with ease of interpretation: if one can understand a flow chart, one can understand a cDFA. On the other hand, cDFAs form a countably infinite concept class with Boolean semantics, and subtle changes to the automaton can result in very different tasks, making them difficult to condition agent behavior on. To address this, we observe that all paths through a DFA correspond to a series of reach-avoid tasks and propose pre-training graph neural network embeddings on "reach-avoid derived" DFAs. Through empirical evaluation, we demonstrate that the proposed pre-training method enables zero-shot generalization to various cDFA task classes and accelerated policy specialization without the myopic suboptimality of hierarchical methods.

Key facts. Cited by 18 papers in the map (data through 7 Oct 2026). Builds on 25 of its 49 references that are in the map. First posted on arXiv 31 Oct 2024.

Builds on

25 of 49 references.

References · 25

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2017Modeling Relational Data with Graph Convolutional Networkscited by 1,684
  3. 2017Hindsight Experience Replaycited by 1,096
  4. 2021How Attentive are Graph Attention Networks?cited by 818
  5. 2016Modular Multitask Reinforcement Learning with Policy Sketchescited by 200
  6. 2020Reward Machines: Exploiting Reward Function Structure in Reinforcement Learningcited by 183
  7. 2019A Survey of Reinforcement Learning Informed by Natural Languagecited by 154
  8. 2019Language as an Abstraction for Hierarchical Deep Reinforcement Learningcited by 138
  9. 2023RoboCLIP: One Demonstration is Enough to Learn Robot Policiescited by 128
  10. 2016Reinforcement Learning With Temporal Logic Rewardscited by 114
  11. 2018Learning to Understand Goal Specifications by Modelling Rewardcited by 80
  12. 2019Control Synthesis from Linear Temporal Logic Specifications using Model-Free Reinforcement Learningcited by 74
13 more
  1. 2021LTL2Action: Generalizing LTL Instructions for Multi-Task RLcited by 72
  2. 2020A Composable Specification Language for Reinforcement Learning Taskscited by 68
  3. 2016Q-Learning for Robust Satisfaction of Signal Temporal Logic Specificationscited by 65
  4. 2021Compositional Reinforcement Learning from Logical Specificationscited by 65
  5. 2014Probably Approximately Correct MDP Learning and Control With Temporal Logic Constraintscited by 53
  6. 2014A Learning Based Approach to Control Synthesis of Markov Decision Processes for Linear Temporal Logic Specificationscited by 49
  7. 2022Policy Optimization with Linear Temporal Logic Constraintscited by 19
  8. 2021On the (In)Tractability of Reinforcement Learning for LTL Objectivescited by 18
  9. 2020Optimal Probabilistic Motion Planning with Potential Infeasible LTL Constraintscited by 14
  10. 2023Policy Synthesis and Reinforcement Learning for Discounted LTLcited by 14
  11. 2023A PAC Learning Algorithm for LTL and Omega-regular Objectives in MDPscited by 12
  12. 2018Finite LTL Synthesis with Environment Assumptions and Quality Measurescited by 4
  13. 2021Demonstration Informed Specification Searchcited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 9; 2026: 9.

18 papers.

Sort:

By area:

  1. 2025Spot the Fake: Large Multimodal Model-Based Synthetic Image Detection with Artifact Explanationcited by 66
  2. 2025One Subgoal at a Time: Zero-Shot Generalization to Arbitrary Linear Temporal Logic Requirements in Multi-Task Reinforcement Learningcited by 8
  3. 2025Ground-Compose-Reinforce: Grounding Language in Agentic Behaviours using Limited Datacited by 5
  4. 2025ARM-FM: Automated Reward Machines via Foundation Models for Compositional Reinforcement Learningcited by 5
  5. 2025Provably Correct Automata Embeddings for Optimal Automata-Conditioned Reinforcement Learningcited by 4
  6. 2026PlatoLTL: Scaling LTL-Guided Multi-Task RLcited by 2
  7. 2025Accelerated Learning with Linear Temporal Logic using Differentiable Simulationcited by 1
  8. 2025Automata-Conditioned Cooperative Multi-Agent Reinforcement Learningcited by 1
  9. 2026Zero-Shot Instruction Following in RL via Structured LTL Representationscited by 1
  10. 2026SpecRLBench: A Benchmark for Generalization in Specification-Guided Reinforcement Learningcited by 1
  11. 2026Controllability in preference-conditioned multi-objective reinforcement learningcited by 1
  12. 2025RLAF: Reinforcement Learning from Automaton Feedbackcited by 0
6 more
  1. 2025Zero-Shot Instruction Following in RL via Structured LTL Representationscited by 0
  2. 2026Learning Bilevel Policies over Symbolic World Models for Long-Horizon Planningcited by 0
  3. 2026Goal-Conditioned Agents that Learn Everything All at Oncecited by 0
  4. 2026Direction-Conditioned Policies via Compositional Subgoal Scoring for Online Goal-Conditioned Reinforcement Learningcited by 0
  5. 2026Temporal Logic Guided Universal Task Representations for Reinforcement Learningcited by 0
  6. 2026Jaxolotl: A Unified High-Performance Benchmark Suite for LTL-Based Multi-Task RLcited by 0