# SafeManip: A Property-Driven Benchmark for Temporal Safety Evaluation in Robotic Manipulation

arXiv: https://arxiv.org/abs/2605.12386 · HTML: https://mlsanity.com/paper/2605.12386/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 12 May 2026.

## Built on by (most cited first, 6 of 6)

- [SafeVLA-Bench: A Benchmark for the Success-Safety Gap in Vision-Language-Action Models](https://mlsanity.com/paper/2606.00773/) (2026)
- [Coding Agents with an Obstacle-Aware Harness for Safe Robot Manipulation](https://mlsanity.com/paper/2609.20822/) (2026)
- [SoftVTBench: A Safety-Aware Visuo-Tactile Benchmark for Physically Constrained Robotic Manipulation of Deformable Objects (Early Version)](https://mlsanity.com/paper/2607.04234/) (2026)
- [No Free Checker: A Survey of Verifiers for Robot Policies](https://mlsanity.com/paper/2609.09250/) (2026)
- [Security and Privacy in Large-Model-Driven Embodied Agents: Attacks, Defenses, and Future Directions](https://mlsanity.com/paper/2609.27847/) (2026)
- [HygieneRoboBench: Benchmarking Hygiene-Aware Planning for Household Robots](https://mlsanity.com/paper/2610.08642/) (2026)

## Abstract (arXiv)

Robotic manipulation is typically evaluated by task success, but reaching the correct final state does not guarantee safe execution. A robot may succeed despite contamination, premature release, or incorrect sequencing. These failures are poorly captured by task-completion metrics or isolated state checks because safety often depends on how behavior unfolds over time. We introduce SafeManip, a benchmark for evaluating temporal safety directly from low-level manipulation rollouts across eight physical and semantic categories. SafeManip grounds symbolic manipulation predicates into reusable temporal properties over finite executions, formalized with LTLf. We further introduce ManipVerse, a property ontology enabling semantic safety rules generalize across objects, tasks, and environments. Finally, SafeManip introduces per-trigger safety metrics that normalize violations by the safety-relevant opportunities that activate each obligation, accounting for differences in how often those opportunities arise across policies and tasks. We instantiate SafeManip on RoboCasa365 and LIBERO to audit seven state-of-the-art robot foundation-model policies. Our results reveal a gap between task capability and temporal safety: higher task success does not necessarily imply safer execution, successful rollouts can still contain violations, and failure patterns vary systematically across safety categories, task horizons, and manipulation suites. SafeManip provides a reusable layer for evaluating how safely manipulation tasks are executed, not only whether they are completed.