Self-Evolving Just-In-Time Memory for Proactive Embodied Safety
On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models
also in 2 more
Abstract (arXiv)
While Vision-Language Models (VLMs) have empowered embodied agents to execute complex household tasks, they struggle to proactively handle dynamically emerging hazards during closed-loop interactions. Existing safety approaches often rely on runtime guardrails to block unsafe actions or induce excessive caution, which severely stalls task progress instead of actively resolving the underlying risks. To break this safety-progress trade-off, we introduce the Self-Evolving Just-In-Time Memory framework, which reframes embodied safety from progress-stalling guardrails to proactive hazard mitigation. The framework consists of a Risk-Sufficient Topological Belief Graph (RSG) for persistent safety-relevant state tracking under partial observability, an Agency-Grounded Factual Memory for precise hazard anticipation, and an Experience Memory that injects procedural Meta-Skills to guide executable, progress-preserving mitigation. Furthermore, we propose an automated Test-Verify-Write loop, allowing agents to continually refine their mitigation Meta-Skills from execution traces at test time. Experiments on IS-Bench demonstrate that our framework substantially boosts the Safe-Success rate across multiple VLM backbones (e.g., +30.3% on Qwen3-VL-8B), enabling agents to proactively mitigate hazards without stalling task progress. Code is available at https://github.com/DyMessi/JIT-Memory.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 31 of its 38 references that are in the map. First posted on arXiv 26 Jun 2026.
Builds on
31 of 38 references.
Closest ancestors · 5
Same direction, newest first
- 2026HomeGuard: VLM-based Embodied Safeguard for Identifying Contextual Risk in Household Task15 Mar
- 2025AGENTSAFE: Benchmarking the Safety of Embodied Agents on Hazardous Instructions17 Jun
- 2025EmbodiedBench: Comprehensive Benchmarking Multi-modal Large Language Models for Vision-Driven Embodied Agents13 Feb
- 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control28 Jul
- 2023PaLM-E: An Embodied Multimodal Language Model6 Mar
Other references · 26
- 2025Qwen3 Technical Reportcited by 8,195
- 2024GPT-4o System Cardcited by 4,467
- 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,302
- 2023Voyager: An Open-Ended Embodied Agent with Large Language Modelscited by 2,081
- 2025Qwen3 Embedding: Advancing Text Embedding and Reranking Through Foundation Modelscited by 1,175
- 2022ProgPrompt: Generating Situated Robot Task Plans using Large Language Modelscited by 598
- 2025Memory in the Age of AI Agentscited by 285
- 2025Hi Robot: Open-Ended Instruction Following with Hierarchical Vision-Language-Action Modelscited by 216
- 2025AgentSpec: Customizable Runtime Enforcement for Safe and Reliable LLM Agentscited by 196
- 2023Instruct2Act: Mapping Multi-modality Instructions to Robotic Actions with Large Language Modelcited by 115
- 2024SafeAgentBench: A Benchmark for Safe Task Planning of Embodied LLM Agentscited by 102
- 20243D-Mem: 3D Scene Memory for Embodied Exploration and Reasoningcited by 73
14 more
- 2023Plug in the Safety Chip: Enforcing Constraints for LLM-driven Robot Agentscited by 67
- 2024Multimodal Situational Safetycited by 54
- 2024A Survey on Robotics with Foundation Models: toward Embodied AIcited by 53
- 2025IS-Bench: Evaluating Interactive Safety of VLM-Driven Embodied Agents in Daily Household Taskscited by 40
- 2024KARMA: Augmenting Embodied AI Agents with Long-and-short Term Memory Systemscited by 38
- 2023Open-Ended Instructable Embodied Agents with Memory-Augmented Large Language Modelscited by 37
- 2025Towards Robust and Secure Embodied AI: A Survey on Vulnerabilities and Attackscited by 32
- 2024EARBench: Towards Evaluating Physical Risk Awareness for Task Planning of Foundation Model-based Embodied AI Agentscited by 24
- 2025Subtle Risks, Critical Failures: A Framework for Diagnosing Physical Safety of LLMs for Embodied Decision Makingcited by 19
- 2025FreezeVLA: Action-Freezing Attacks against Vision-Language-Action Modelscited by 19
- 2023Modeling Dynamic Environments with Scene Graph Memorycited by 18
- 2024Easier Said Than Done: Unpacking Intent-Behavior Gap in Jailbreaking LLM-Based Robotscited by 14
- 2025RoboMemory: A Brain-inspired Multi-memory Agentic Framework for Interactive Environmental Learning in Physical Embodied Systemscited by 14
- 2025RoboSafe: Safeguarding Embodied Agents via Executable Safety Logiccited by 11
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026Safe Task Planning with Long-Term Graph Memory for Embodied Agents
- 2026SafeRelBench: A Spatial-Relation-Aware Benchmark for Process-Level Safety in VLM-Driven Embodied Agents
- 2025SafeMind: Benchmarking and Mitigating Safety Risks in Embodied LLM Agents
- 2026EMBGuard: Constructing Hazard-Aware Guardrails for Safe Planning in Embodied Agents
- 2026HazardArena: Evaluating Semantic Safety in Vision-Language-Action Models
- 2026GuardianBench: A Same-Scene Instruction-Contrastive Benchmark for Latent Contextual Risk in Embodied AI
- 2026SafeBranch: Branch-Pair Safety Alignment for Embodied Agents
- 2026RT-Safe: Benchmarking Agent Safety in Real-Time Embodied Environment