# Multi-Agent Reinforcement Learning with Temporal Logic Specifications

arXiv: https://arxiv.org/abs/2102.00582 · HTML: https://mlsanity.com/paper/2102.00582/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 64 references that are in the map.
- First posted on arXiv 1 Feb 2021.

## Builds on (most cited first, 23 of 23)

- [Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments](https://mlsanity.com/paper/1706.02275/) (2017)
- [Counterfactual Multi-Agent Policy Gradients](https://mlsanity.com/paper/1705.08926/) (2017)
- [Multi-Agent Reinforcement Learning: A Selective Overview of Theories and Algorithms](https://mlsanity.com/paper/1911.10635/) (2019)
- [Fully Decentralized Multi-Agent Reinforcement Learning with Networked Agents](https://mlsanity.com/paper/1802.08757/) (2018)
- [On the Theory of Policy Gradient Methods: Optimality, Approximation, and Distribution Shift](https://mlsanity.com/paper/1908.00261/) (2019)
- [Reinforcement Learning With Temporal Logic Rewards](https://mlsanity.com/paper/1612.03471/) (2016)
- [Omega-Regular Objectives in Model-Free Reinforcement Learning](https://mlsanity.com/paper/1810.00950/) (2018)
- [Control Synthesis from Linear Temporal Logic Specifications using Model-Free Reinforcement Learning](https://mlsanity.com/paper/1909.07299/) (2019)
- [A Composable Specification Language for Reinforcement Learning Tasks](https://mlsanity.com/paper/2008.09293/) (2020)
- [Environment-Independent Task Specifications via GLTL](https://mlsanity.com/paper/1704.04341/) (2017)
- [Probably Approximately Correct MDP Learning and Control With Temporal Logic Constraints](https://mlsanity.com/paper/1404.7073/) (2014)
- [A Learning Based Approach to Control Synthesis of Markov Decision Processes for Linear Temporal Logic Specifications](https://mlsanity.com/paper/1409.5486/) (2014)
- [Is the Policy Gradient a Gradient?](https://mlsanity.com/paper/1906.07073/) (2019)
- [Verification of Markov Decision Processes using Learning Algorithms](https://mlsanity.com/paper/1402.2967/) (2014)
- [Certified Reinforcement Learning with Logic Guidance](https://mlsanity.com/paper/1902.00778/) (2019)
- [Decentralized Q-Learning for Stochastic Teams and Games](https://mlsanity.com/paper/1506.07924/) (2015)
- [PAC Statistical Model Checking for Markov Decision Processes and Stochastic Games](https://mlsanity.com/paper/1905.04403/) (2019)
- [Logically-Constrained Neural Fitted Q-Iteration](https://mlsanity.com/paper/1809.07823/) (2018)
- [Reinforcement Learning of Control Policy for Linear Temporal Logic Specifications Using Limit-Deterministic Generalized B"uchi Automata](https://mlsanity.com/paper/2001.04669/) (2020)
- [Extended Markov Games to Learn Multiple Tasks in Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2002.06000/) (2020)
- [Equilibria-based Probabilistic Model Checking for Concurrent Stochastic Games](https://mlsanity.com/paper/1811.07145/) (2018)
- [Automata Guided Reinforcement Learning With Demonstrations](https://mlsanity.com/paper/1809.06305/) (2018)
- [Reward Shaping for Reinforcement Learning with Omega-Regular Objectives](https://mlsanity.com/paper/2001.05977/) (2020)

## Built on by (most cited first, 27 of 27)

- [On the Expressivity of Markov Reward](https://mlsanity.com/paper/2111.00876/) (2021)
- [Compositional Reinforcement Learning from Logical Specifications](https://mlsanity.com/paper/2106.13906/) (2021)
- [In a Nutshell, the Human Asked for This: Latent Goals for Following Temporal Specifications](https://mlsanity.com/paper/2110.09461/) (2021)
- [Three Dogmas of Reinforcement Learning](https://mlsanity.com/paper/2407.10583/) (2024)
- [LgTS: Dynamic Task Sampling using LLM-generated sub-goals for Reinforcement Learning Agents](https://mlsanity.com/paper/2310.09454/) (2023)
- [Specification-Guided Learning of Nash Equilibria with High Social Welfare](https://mlsanity.com/paper/2206.03348/) (2022)
- [Multi-Agent Reinforcement Learning Guided by Signal Temporal Logic Specifications](https://mlsanity.com/paper/2306.06808/) (2023)
- [Model-Free Learning of Safe yet Effective Controllers](https://mlsanity.com/paper/2103.14600/) (2021)
- [Reinforcement Learning with Knowledge Representation and Reasoning: A Brief Survey](https://mlsanity.com/paper/2304.12090/) (2023)
- [(Ir)rationality in AI: State of the Art, Research Challenges and Open Questions](https://mlsanity.com/paper/2311.17165/) (2023)
- [Standpoint Linear Temporal Logic](https://mlsanity.com/paper/2304.14243/) (2023)
- [Assume-Guarantee Reinforcement Learning](https://mlsanity.com/paper/2312.09938/) (2023)
- [DistSPECTRL: Distributing Specifications in Multi-Agent Reinforcement Learning Systems](https://mlsanity.com/paper/2206.13754/) (2022)
- [Logical Specifications-guided Dynamic Task Sampling for Reinforcement Learning Agents](https://mlsanity.com/paper/2402.03678/) (2024)
- [Tractable Offline Learning of Regular Decision Processes](https://mlsanity.com/paper/2409.02747/) (2024)
- [Distributed Control using Reinforcement Learning with Temporal-Logic-Based Reward Shaping](https://mlsanity.com/paper/2203.04172/) (2022)
- [Optimal Control of Logically Constrained Partially Observable and Multi-Agent Markov Decision Processes](https://mlsanity.com/paper/2305.14736/) (2023)
- [HypRL: Reinforcement Learning of Control Policies for Hyperproperties](https://mlsanity.com/paper/2504.04675/) (2025)
- [PlatoLTL: Scaling LTL-Guided Multi-Task RL](https://mlsanity.com/paper/2601.22891/) (2026)
- [Multi-Agent Verification and Control with Probabilistic Model Checking](https://mlsanity.com/paper/2308.02829/) (2023)
- [Automata-Conditioned Cooperative Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2511.02304/) (2025)
- [SpecRLBench: A Benchmark for Generalization in Specification-Guided Reinforcement Learning](https://mlsanity.com/paper/2604.24729/) (2026)
- [CatlNet: Learning Communication and Coordination Policies from CaTL+ Specifications](https://mlsanity.com/paper/2212.11792/) (2022)
- [Adversarial Reinforcement Learning for Offensive and Defensive Agents in a Simulated Zero-Sum Network Environment](https://mlsanity.com/paper/2510.05157/) (2025)
- [HyPOLE: Hyperproperty-Guided Multi-Agent Reinforcement Learning under Partial Observation](https://mlsanity.com/paper/2606.30966/) (2026)
- [The syntax and semantics of goals](https://mlsanity.com/paper/2609.19448/) (2026)
- [Epistemic-Probabilistic Model for Guarded Multi-Agent LLM Coordination](https://mlsanity.com/paper/2609.29366/) (2026)

## Abstract (arXiv)

In this paper, we study the problem of learning to satisfy temporal logic specifications with a group of agents in an unknown environment, which may exhibit probabilistic behaviour. From a learning perspective these specifications provide a rich formal language with which to capture tasks or objectives, while from a logic and automated verification perspective the introduction of learning capabilities allows for practical applications in large, stochastic, unknown environments. The existing work in this area is, however, limited. Of the frameworks that consider full linear temporal logic or have correctness guarantees, all methods thus far consider only the case of a single temporal logic specification and a single agent. In order to overcome this limitation, we develop the first multi-agent reinforcement learning technique for temporal logic specifications, which is also novel in its ability to handle multiple specifications. We provide correctness and convergence guarantees for our main algorithm - ALMANAC (Automaton/Logic Multi-Agent Natural Actor-Critic) - even when using function approximation. Alongside our theoretical results, we further demonstrate the applicability of our technique via a set of preliminary experiments.