# Safe Reinforcement Learning with Natural Language Constraints

arXiv: https://arxiv.org/abs/2010.05150 · HTML: https://mlsanity.com/paper/2010.05150/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 11 Oct 2020.

## Built on by (most cited first, 23 of 23)

- [FLIRT: Feedback Loop In-context Red Teaming](https://mlsanity.com/paper/2308.04265/) (2023)
- [A Survey of Constraint Formulations in Safe Reinforcement Learning](https://mlsanity.com/paper/2402.02025/) (2024)
- [LLF-Bench: Benchmark for Interactive Learning from Language Feedback](https://mlsanity.com/paper/2312.06853/) (2023)
- [Leveraging Language for Accelerated Learning of Tool Manipulation](https://mlsanity.com/paper/2206.13074/) (2022)
- [Controllable Text Generation with Language Constraints](https://mlsanity.com/paper/2212.10466/) (2022)
- [GLUECons: A Generic Benchmark for Learning Under Constraints](https://mlsanity.com/paper/2302.10914/) (2023)
- [Embedding-Aligned Language Models](https://mlsanity.com/paper/2406.00024/) (2024)
- [Safe Multi-agent Reinforcement Learning with Natural Language Constraints](https://mlsanity.com/paper/2405.20018/) (2024)
- [SAFER: Data-Efficient and Safe Reinforcement Learning via Skill Acquisition](https://mlsanity.com/paper/2202.04849/) (2022)
- [One-Shot Learning from a Demonstration with Hierarchical Latent Language](https://mlsanity.com/paper/2203.04806/) (2022)
- [Architext: Language-Driven Generative Architecture Design](https://mlsanity.com/paper/2303.07519/) (2023)
- [Reinforcement Learning with Knowledge Representation and Reasoning: A Brief Survey](https://mlsanity.com/paper/2304.12090/) (2023)
- [Mutual Enhancement of Large Language and Reinforcement Learning Models through Bi-Directional Feedback Mechanisms: A Planning Case Study](https://mlsanity.com/paper/2401.06603/) (2024)
- [A Reminder of its Brittleness: Language Reward Shaping May Hinder Learning for Instruction Following Agents](https://mlsanity.com/paper/2305.16621/) (2023)
- [Offline-to-online Reinforcement Learning for Image-based Grasping with Scarce Demonstrations](https://mlsanity.com/paper/2410.14957/) (2024)
- [Learning Natural Language Constraints for Safe Reinforcement Learning of Language Agents](https://mlsanity.com/paper/2504.03185/) (2025)
- [A Computational Interface to Translate Strategic Intent from Unstructured Language in a Low-Data Setting](https://mlsanity.com/paper/2208.08374/) (2022)
- [Modeling Risk in Reinforcement Learning: A Literature Mapping](https://mlsanity.com/paper/2312.05231/) (2023)
- [SpecRLBench: A Benchmark for Generalization in Specification-Guided Reinforcement Learning](https://mlsanity.com/paper/2604.24729/) (2026)
- [Output-Constrained Decision Trees](https://mlsanity.com/paper/2405.15314/) (2024)
- [Towards Online Safety Corrections for Robotic Manipulation Policies](https://mlsanity.com/paper/2409.08233/) (2024)
- [From Text to Trajectory: Exploring Complex Constraint Representation and Decomposition in Safe Reinforcement Learning](https://mlsanity.com/paper/2412.08920/) (2024)
- [ShieldVLA: Feasibility-Aware Safety Alignment for Vision-Language-Action Models](https://mlsanity.com/paper/2609.13231/) (2026)

## Abstract (arXiv)

While safe reinforcement learning (RL) holds great promise for many practical applications like robotics or autonomous cars, current approaches require specifying constraints in mathematical form. Such specifications demand domain expertise, limiting the adoption of safe RL. In this paper, we propose learning to interpret natural language constraints for safe RL. To this end, we first introduce HazardWorld, a new multi-task benchmark that requires an agent to optimize reward while not violating constraints specified in free-form text. We then develop an agent with a modular architecture that can interpret and adhere to such textual constraints while learning new tasks. Our model consists of (1) a constraint interpreter that encodes textual constraints into spatial and temporal representations of forbidden states, and (2) a policy network that uses these representations to produce a policy achieving minimal constraint violations during training. Across different domains in HazardWorld, we show that our method achieves higher rewards (up to11x) and fewer constraint violations (by 1.8x) compared to existing approaches. However, in terms of absolute performance, HazardWorld still poses significant challenges for agents to learn efficiently, motivating the need for future work.