# Updating Robot Safety Representations Online from Natural Language Feedback

arXiv: https://arxiv.org/abs/2409.14580 · HTML: https://mlsanity.com/paper/2409.14580/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 22 Sep 2024.

## Built on by (most cited first, 8 of 8)

- [Generating Robot Constitutions & Benchmarks for Semantic Safety](https://mlsanity.com/paper/2503.08663/) (2025)
- [Real-Time Out-of-Distribution Failure Prevention via Multi-Modal Reasoning](https://mlsanity.com/paper/2505.10547/) (2025)
- [AnySafe: Adapting Latent Safety Filters at Runtime via Safety Constraint Parameterization in the Latent Space](https://mlsanity.com/paper/2509.19555/) (2025)
- [CARE: Enhancing Safety of Visual Navigation through Collision Avoidance via Repulsive Estimation](https://mlsanity.com/paper/2506.03834/) (2025)
- [Your Model Already Knows: Attention-Guided Safety Filter for Vision-Language-Action Models](https://mlsanity.com/paper/2606.09749/) (2026)
- [Towards General Language-Conditioned Latent Safety Filters](https://mlsanity.com/paper/2608.00315/) (2026)
- [Vision--Language Signals in Constrained RL: Safety Gains Without Anticipation](https://mlsanity.com/paper/2609.34041/) (2026)
- [Seeing Before Colliding: Anticipatory Safe RL with Frozen Vision-Language Models](https://mlsanity.com/paper/2606.11266/) (2026)

## Abstract (arXiv)

Robots must operate safely when deployed in novel and human-centered environments, like homes. Current safe control approaches typically assume that the safety constraints are known a priori, and thus, the robot can pre-compute a corresponding safety controller. While this may make sense for some safety constraints (e.g., avoiding collision with walls by analyzing a floor plan), other constraints are more complex (e.g., spills), inherently personal, context-dependent, and can only be identified at deployment time when the robot is interacting in a specific environment and with a specific person (e.g., fragile objects, expensive rugs). Here, language provides a flexible mechanism to communicate these evolving safety constraints to the robot. In this work, we use vision language models (VLMs) to interpret language feedback and the robot's image observations to continuously update the robot's representation of safety constraints. With these inferred constraints, we update a Hamilton-Jacobi reachability safety controller online via efficient warm-starting techniques. Through simulation and hardware experiments, we demonstrate the robot's ability to infer and respect language-based safety constraints with the proposed approach.