# Encoding formulas as deep networks: Reinforcement learning for zero-shot execution of LTL formulas

arXiv: https://arxiv.org/abs/2006.01110 · HTML: https://mlsanity.com/paper/2006.01110/

## Key facts

- Cited by 36 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 1 Jun 2020.

## Built on by (most cited first, 30 of 36)

- [LTL2Action: Generalizing LTL Instructions for Multi-Task RL](https://mlsanity.com/paper/2102.06858/) (2021)
- [Compositional Reinforcement Learning from Logical Specifications](https://mlsanity.com/paper/2106.13906/) (2021)
- [The Logical Options Framework](https://mlsanity.com/paper/2102.12571/) (2021)
- [DeepLTL: Learning to Efficiently Satisfy Complex LTL Specifications for Multi-Task RL](https://mlsanity.com/paper/2410.04631/) (2024)
- [In a Nutshell, the Human Asked for This: Latent Goals for Following Temporal Specifications](https://mlsanity.com/paper/2110.09461/) (2021)
- [General agents contain world models](https://mlsanity.com/paper/2506.01622/) (2025)
- [BlendRL: A Framework for Merging Symbolic and Neural Policy Learning](https://mlsanity.com/paper/2410.11689/) (2024)
- [Learning to Follow Instructions in Text-Based Games](https://mlsanity.com/paper/2211.04591/) (2022)
- [Reward Machines for Deep RL in Noisy and Uncertain Environments](https://mlsanity.com/paper/2406.00120/) (2024)
- [Exploiting Transformer in Sparse Reward Reinforcement Learning for Interpretable Temporal Logic Motion Planning](https://mlsanity.com/paper/2209.13220/) (2022)
- [Noisy Symbolic Abstractions for Deep RL: A case study with Reward Machines](https://mlsanity.com/paper/2211.10902/) (2022)
- [Neural Reward Machines](https://mlsanity.com/paper/2408.08677/) (2024)
- [Reinforcement Learning with Knowledge Representation and Reasoning: A Brief Survey](https://mlsanity.com/paper/2304.12090/) (2023)
- [Ground-Compose-Reinforce: Grounding Language in Agentic Behaviours using Limited Data](https://mlsanity.com/paper/2507.10741/) (2025)
- [Robust Subtask Learning for Compositional Generalization](https://mlsanity.com/paper/2302.02984/) (2023)
- [Planning with a Learned Policy Basis to Optimally Solve Complex Tasks](https://mlsanity.com/paper/2403.15301/) (2024)
- [Guiding Multi-agent Multi-task Reinforcement Learning by a Hierarchical Framework with Logical Reward Shaping](https://mlsanity.com/paper/2411.01184/) (2024)
- [Reinforcement Learning with Physics-Informed Symbolic Program Priors for Zero-Shot Wireless Indoor Navigation](https://mlsanity.com/paper/2506.22365/) (2025)
- [Logic-based Reward Shaping for Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2206.08881/) (2022)
- [Inductive Generalization in Reinforcement Learning from Specifications](https://mlsanity.com/paper/2406.03651/) (2024)
- [Exploiting Hybrid Policy in Reinforcement Learning for Interpretable Temporal Logic Manipulation](https://mlsanity.com/paper/2412.20338/) (2024)
- [HypRL: Reinforcement Learning of Control Policies for Hyperproperties](https://mlsanity.com/paper/2504.04675/) (2025)
- [Beyond Fixed Tasks: Unsupervised Environment Design for Task-Level Pairs](https://mlsanity.com/paper/2511.12706/) (2025)
- [Generalizing LTL Instructions via Future Dependent Options](https://mlsanity.com/paper/2212.04576/) (2022)
- [Neural Controller Synthesis for Signal Temporal Logic Specifications Using Encoder-Decoder Structured Networks](https://mlsanity.com/paper/2212.05200/) (2022)
- [Reward-Machine-Guided, Self-Paced Reinforcement Learning](https://mlsanity.com/paper/2305.16505/) (2023)
- [Generalization of Compositional Tasks with Logical Specification via Implicit Planning](https://mlsanity.com/paper/2410.09686/) (2024)
- [Fully Learnable Neural Reward Machines](https://mlsanity.com/paper/2509.19017/) (2025)
- [Grounding LTL Tasks in Sub-Symbolic RL Environments for Zero-Shot Generalization](https://mlsanity.com/paper/2602.09761/) (2026)
- [Zero-Shot Instruction Following in RL via Structured LTL Representations](https://mlsanity.com/paper/2602.14344/) (2026)

## Abstract (arXiv)

We demonstrate a reinforcement learning agent which uses a compositional recurrent neural network that takes as input an LTL formula and determines satisfying actions. The input LTL formulas have never been seen before, yet the network performs zero-shot generalization to satisfy them. This is a novel form of multi-task learning for RL agents where agents learn from one diverse set of tasks and generalize to a new set of diverse tasks. The formulation of the network enables this capacity to generalize. We demonstrate this ability in two domains. In a symbolic domain, the agent finds a sequence of letters that is accepted. In a Minecraft-like environment, the agent finds a sequence of actions that conform to the formula. While prior work could learn to execute one formula reliably given examples of that formula, we demonstrate how to encode all formulas reliably. This could form the basis of new multitask agents that discover sub-tasks and execute them without any additional training, as well as the agents which follow more complex linguistic commands. The structures required for this generalization are specific to LTL formulas, which opens up an interesting theoretical question: what structures are required in neural networks for zero-shot generalization to different logics?