mlsanityData to 7 Oct 2026

Self-Evolving Just-In-Time Memory for Proactive Embodied Safety

arXiv 2607.16247 ↗First posted 26 Jun 2026cs.LG cs.CVcited by 0

On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models

also in 2 more
Abstract (arXiv)

While Vision-Language Models (VLMs) have empowered embodied agents to execute complex household tasks, they struggle to proactively handle dynamically emerging hazards during closed-loop interactions. Existing safety approaches often rely on runtime guardrails to block unsafe actions or induce excessive caution, which severely stalls task progress instead of actively resolving the underlying risks. To break this safety-progress trade-off, we introduce the Self-Evolving Just-In-Time Memory framework, which reframes embodied safety from progress-stalling guardrails to proactive hazard mitigation. The framework consists of a Risk-Sufficient Topological Belief Graph (RSG) for persistent safety-relevant state tracking under partial observability, an Agency-Grounded Factual Memory for precise hazard anticipation, and an Experience Memory that injects procedural Meta-Skills to guide executable, progress-preserving mitigation. Furthermore, we propose an automated Test-Verify-Write loop, allowing agents to continually refine their mitigation Meta-Skills from execution traces at test time. Experiments on IS-Bench demonstrate that our framework substantially boosts the Safe-Success rate across multiple VLM backbones (e.g., +30.3% on Qwen3-VL-8B), enabling agents to proactively mitigate hazards without stalling task progress. Code is available at https://github.com/DyMessi/JIT-Memory.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 31 of its 38 references that are in the map. First posted on arXiv 26 Jun 2026.

Builds on

31 of 38 references.

Closest ancestors · 5

Same direction, newest first

  1. 2026HomeGuard: VLM-based Embodied Safeguard for Identifying Contextual Risk in Household Task15 Mar
  2. 2025AGENTSAFE: Benchmarking the Safety of Embodied Agents on Hazardous Instructions17 Jun
  3. 2025EmbodiedBench: Comprehensive Benchmarking Multi-modal Large Language Models for Vision-Driven Embodied Agents13 Feb
  4. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control28 Jul
  5. 2023PaLM-E: An Embodied Multimodal Language Model6 Mar

Other references · 26

  1. 2025Qwen3 Technical Reportcited by 8,195
  2. 2024GPT-4o System Cardcited by 4,467
  3. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,302
  4. 2023Voyager: An Open-Ended Embodied Agent with Large Language Modelscited by 2,081
  5. 2025Qwen3 Embedding: Advancing Text Embedding and Reranking Through Foundation Modelscited by 1,175
  6. 2022ProgPrompt: Generating Situated Robot Task Plans using Large Language Modelscited by 598
  7. 2025Memory in the Age of AI Agentscited by 285
  8. 2025Hi Robot: Open-Ended Instruction Following with Hierarchical Vision-Language-Action Modelscited by 216
  9. 2025AgentSpec: Customizable Runtime Enforcement for Safe and Reliable LLM Agentscited by 196
  10. 2023Instruct2Act: Mapping Multi-modality Instructions to Robotic Actions with Large Language Modelcited by 115
  11. 2024SafeAgentBench: A Benchmark for Safe Task Planning of Embodied LLM Agentscited by 102
  12. 20243D-Mem: 3D Scene Memory for Embodied Exploration and Reasoningcited by 73
14 more
  1. 2023Plug in the Safety Chip: Enforcing Constraints for LLM-driven Robot Agentscited by 67
  2. 2024Multimodal Situational Safetycited by 54
  3. 2024A Survey on Robotics with Foundation Models: toward Embodied AIcited by 53
  4. 2025IS-Bench: Evaluating Interactive Safety of VLM-Driven Embodied Agents in Daily Household Taskscited by 40
  5. 2024KARMA: Augmenting Embodied AI Agents with Long-and-short Term Memory Systemscited by 38
  6. 2023Open-Ended Instructable Embodied Agents with Memory-Augmented Large Language Modelscited by 37
  7. 2025Towards Robust and Secure Embodied AI: A Survey on Vulnerabilities and Attackscited by 32
  8. 2024EARBench: Towards Evaluating Physical Risk Awareness for Task Planning of Foundation Model-based Embodied AI Agentscited by 24
  9. 2025Subtle Risks, Critical Failures: A Framework for Diagnosing Physical Safety of LLMs for Embodied Decision Makingcited by 19
  10. 2025FreezeVLA: Action-Freezing Attacks against Vision-Language-Action Modelscited by 19
  11. 2023Modeling Dynamic Environments with Scene Graph Memorycited by 18
  12. 2024Easier Said Than Done: Unpacking Intent-Behavior Gap in Jailbreaking LLM-Based Robotscited by 14
  13. 2025RoboMemory: A Brain-inspired Multi-memory Agentic Framework for Interactive Environmental Learning in Physical Embodied Systemscited by 14
  14. 2025RoboSafe: Safeguarding Embodied Agents via Executable Safety Logiccited by 11

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Safe Task Planning with Long-Term Graph Memory for Embodied Agents
  2. 2026SafeRelBench: A Spatial-Relation-Aware Benchmark for Process-Level Safety in VLM-Driven Embodied Agents
  3. 2025SafeMind: Benchmarking and Mitigating Safety Risks in Embodied LLM Agents
  4. 2026EMBGuard: Constructing Hazard-Aware Guardrails for Safe Planning in Embodied Agents
  5. 2026HazardArena: Evaluating Semantic Safety in Vision-Language-Action Models
  6. 2026GuardianBench: A Same-Scene Instruction-Contrastive Benchmark for Latent Contextual Risk in Embodied AI
  7. 2026SafeBranch: Branch-Pair Safety Alignment for Embodied Agents
  8. 2026RT-Safe: Benchmarking Agent Safety in Real-Time Embodied Environment