mlsanityData to 14 Oct 2026

Safe Multi-agent Reinforcement Learning with Natural Language Constraints

arXiv 2405.20018 ↗First posted 30 May 2024cs.MA cs.CL cs.LGcited by 7

On the map: LLM agents and tool use → Reinforcement learning for LLM agents

Abstract (arXiv)

The role of natural language constraints in Safe Multi-agent Reinforcement Learning (MARL) is crucial, yet often overlooked. While Safe MARL has vast potential, especially in fields like robotics and autonomous vehicles, its full potential is limited by the need to define constraints in pre-designed mathematical terms, which requires extensive domain expertise and reinforcement learning knowledge, hindering its broader adoption. To address this limitation and make Safe MARL more accessible and adaptable, we propose a novel approach named Safe Multi-agent Reinforcement Learning with Natural Language constraints (SMALL). Our method leverages fine-tuned language models to interpret and process free-form textual constraints, converting them into semantic embeddings that capture the essence of prohibited states and behaviours. These embeddings are then integrated into the multi-agent policy learning process, enabling agents to learn policies that minimize constraint violations while optimizing rewards. To evaluate the effectiveness of SMALL, we introduce the LaMaSafe, a multi-task benchmark designed to assess the performance of multiple agents in adhering to natural language constraints. Empirical evaluations across various environments demonstrate that SMALL achieves comparable rewards and significantly fewer constraint violations, highlighting its effectiveness in understanding and enforcing natural language constraints.

Key facts. Cited by 7 papers in the map (data through 14 Oct 2026). Placed in Reinforcement learning for LLM agents, one of 16 directions in LLM agents and tool use. Builds on 19 of its 72 references that are in the map. First posted on arXiv 30 May 2024.

Builds on

19 of 72 references.

References · 19

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2017Proximal Policy Optimization Algorithmscited by 15,215
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  6. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  7. 2021The Surprising Effectiveness of PPO in Cooperative, Multi-Agent Gamescited by 942
  8. 2023AgentVerse: Facilitating Multi-Agent Collaboration and Exploring Emergent Behaviorscited by 598
  9. 2023Minigrid & Miniworld: Modular & Customizable Reinforcement Learning Environments for Goal-Oriented Taskscited by 327
  10. 2021Trust Region Policy Optimisation in Multi-Agent Reinforcement Learningcited by 186
  11. 2022A Review of Safe Reinforcement Learning: Methods, Theory and Applicationscited by 181
  12. 2020FACMAC: Factored Multi-Agent Centralised Policy Gradientscited by 178
7 more
  1. 2023Waymax: An Accelerated, Data-Driven Simulator for Large-Scale Autonomous Driving Researchcited by 160
  2. 2023Safety-Gymnasium: A Unified Safe Reinforcement Learning Benchmarkcited by 143
  3. 2021Safe Multi-Agent Reinforcement Learning via Shieldingcited by 61
  4. 2021Multi-Agent Constrained Policy Optimisationcited by 35
  5. 2020Safe Reinforcement Learning with Natural Language Constraintscited by 23
  6. 2019A Cooperative Multi-Agent Reinforcement Learning Framework for Resource Balancing in Complex Logistics Networkcited by 22
  7. 2021Safe Multi-Agent Reinforcement Learning through Decentralized Multiple Control Barrier Functionscited by 10

Built on by

Cited by year

Citing papers per year: 2024: 3; 2025: 4.

7 papers.

  1. 2025M3HF: Multi-agent Reinforcement Learning from Multi-phase Human Feedback of Mixed Qualitycited by 11
  2. 2025Eigenspectrum Analysis of Neural Networks without Aspect Ratio Biascited by 10
  3. 2024YOLO-MARL: You Only LLM Once for Multi-Agent Reinforcement Learningcited by 9
  4. 2025HASARD: A Benchmark for Vision-Based Safe Reinforcement Learning in Embodied Agentscited by 6
  5. 2024MARLIN: Multi-Agent Reinforcement Learning Guided by Language-Based Inter-Robot Negotiationcited by 5
  6. 2025A Constrained Multi-Agent Reinforcement Learning Approach to Autonomous Traffic Signal Controlcited by 3
  7. 2024Think Smart, Act SMARL! Analyzing Probabilistic Logic Shields for Multi-Agent Reinforcement Learningcited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2024Safe Reinforcement Learning with Free-form Natural Language Constraints and Pre-Trained Language Models
  2. 2024From Text to Trajectory: Exploring Complex Constraint Representation and Decomposition in Safe Reinforcement Learning
  3. 2025Learning Natural Language Constraints for Safe Reinforcement Learning of Language Agents
  4. 2025A Survey of Safe Reinforcement Learning and Constrained MDPs: A Technical Survey on Single-Agent and Multi-Agent Safety
  5. 2025Resolving Conflicting Constraints in Multi-Agent Reinforcement Learning with Layered Safety
  6. 2024Natural Language Reinforcement Learning
  7. 2024A Survey of Constraint Formulations in Safe Reinforcement Learning
  8. 2024Safe Multi-Agent Reinforcement Learning with Convergence to Generalized Nash Equilibrium