# Concept-Based Dictionary Learning for Inference-Time Safety in Vision Language Action Models

arXiv: https://arxiv.org/abs/2602.01834 · HTML: https://mlsanity.com/paper/2602.01834/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Placed in Safety and adversarial attacks, one of 13 directions in Vision-language-action models.
- Builds on 41 of its 62 references that are in the map.
- First posted on arXiv 2 Feb 2026.

## Placed in

- Vision-language-action models / [Safety and adversarial attacks](https://mlsanity.com/vla/safety-and-adversarial-attacks/)

## Builds on (most cited first, 30 of 41)

- [RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control](https://mlsanity.com/paper/2307.15818/) (2023)
- [OpenVLA: An Open-Source Vision-Language-Action Model](https://mlsanity.com/paper/2406.09246/) (2024)
- [PaLM-E: An Embodied Multimodal Language Model](https://mlsanity.com/paper/2303.03378/) (2023)
- [Octo: An Open-Source Generalist Robot Policy](https://mlsanity.com/paper/2405.12213/) (2024)
- [Code as Policies: Language Model Programs for Embodied Control](https://mlsanity.com/paper/2209.07753/) (2022)
- [VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Models](https://mlsanity.com/paper/2307.05973/) (2023)
- [RDT-1B: a Diffusion Foundation Model for Bimanual Manipulation](https://mlsanity.com/paper/2410.07864/) (2024)
- [Safe RLHF: Safe Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2310.12773/) (2023)
- [CLIPort: What and Where Pathways for Robotic Manipulation](https://mlsanity.com/paper/2109.12098/) (2021)
- [SmoothLLM: Defending Large Language Models Against Jailbreaking Attacks](https://mlsanity.com/paper/2310.03684/) (2023)
- [UniVLA: Learning to Act Anywhere with Task-centric Latent Actions](https://mlsanity.com/paper/2505.06111/) (2025)
- [VIMA: General Robot Manipulation with Multimodal Prompts](https://mlsanity.com/paper/2210.03094/) (2022)
- [DreamVLA: A Vision-Language-Action Model Dreamed with Comprehensive World Knowledge](https://mlsanity.com/paper/2507.04447/) (2025)
- [DexVLA: Vision-Language Model with Plug-In Diffusion Expert for General Robot Control](https://mlsanity.com/paper/2502.05855/) (2025)
- [AdaShield: Safeguarding Multimodal Large Language Models from Structure-based Attack via Adaptive Shield Prompting](https://mlsanity.com/paper/2403.09513/) (2024)
- [MLLM-Protector: Ensuring MLLM's Safety without Hurting Performance](https://mlsanity.com/paper/2401.02906/) (2024)
- [CoVLA: Comprehensive Vision-Language-Action Dataset for Autonomous Driving](https://mlsanity.com/paper/2408.10845/) (2024)
- [Gradient Cuff: Detecting Jailbreak Attacks on Large Language Models by Exploring Refusal Loss Landscapes](https://mlsanity.com/paper/2403.00867/) (2024)
- [JailGuard: A Universal Detection Framework for LLM Prompt-based Attacks](https://mlsanity.com/paper/2312.10766/) (2023)
- [IS-Bench: Evaluating Interactive Safety of VLM-Driven Embodied Agents in Daily Household Tasks](https://mlsanity.com/paper/2506.16402/) (2025)
- [Understanding the Repeat Curse in Large Language Models from a Feature Perspective](https://mlsanity.com/paper/2504.14218/) (2025)
- [PARDEN, Can You Repeat That? Defending against Jailbreaks via Repetition](https://mlsanity.com/paper/2405.07932/) (2024)
- [Curriculum-RLAIF: Curriculum Alignment with Reinforcement Learning from AI Feedback](https://mlsanity.com/paper/2505.20075/) (2025)
- [BlueSuffix: Reinforced Blue Teaming for Vision-Language Models Against Jailbreak Attacks](https://mlsanity.com/paper/2410.20971/) (2024)
- [Safeguarding Vision-Language Models Against Patched Visual Prompt Injectors](https://mlsanity.com/paper/2405.10529/) (2024)
- [Exploring the Personality Traits of LLMs through Latent Features Steering](https://mlsanity.com/paper/2410.10863/) (2024)
- [Model-agnostic Adversarial Attack and Defense for Vision-Language-Action Models](https://mlsanity.com/paper/2510.13237/) (2025)
- [ObjectVLA: End-to-End Open-World Object Manipulation Without Demonstration](https://mlsanity.com/paper/2502.19250/) (2025)
- [Goal-oriented Backdoor Attack against Vision-Language-Action Models via Physical Objects](https://mlsanity.com/paper/2510.09269/) (2025)
- [To Protect the LLM Agent Against the Prompt Injection Attack with Polymorphic Prompt](https://mlsanity.com/paper/2506.05739/) (2025)

## Built on by (most cited first, 3 of 3)

- [Embodied Interpretability: Linking Causal Understanding to Generalization in Vision-Language-Action Models](https://mlsanity.com/paper/2605.00321/) (2026)
- [Security of World-Model-Based Embodied AI: A Lifecycle of Threats, Defenses, and Evaluation](https://mlsanity.com/paper/2607.28226/) (2026)
- [Safe Task Planning with Long-Term Graph Memory for Embodied Agents](https://mlsanity.com/paper/2609.08444/) (2026)

## Abstract (arXiv)

Vision Language Action (VLA) models close the perception action loop by translating multimodal instructions into executable behaviors, but this very capability magnifies safety risks: jailbreaks that merely yield toxic text in LLMs can trigger unsafe physical actions in embodied systems. Existing defenses alignment, filtering, or prompt hardening intervene too late or at the wrong modality, leaving fused representations exploitable. We introduce a concept based dictionary learning framework for inference time safety control. By learning sparse, interpretable dictionaries from hidden activations, our method identifies harmful concept directions and attenuates risky components when the estimated risk exceeds a threshold. Experiments on Libero-Harm, BadRobot, RoboPair, and IS-Bench show that our approach achieves state-of-the-art defense performance, cutting attack success rates by over 70% while maintaining task success. Crucially, the framework is plug-in and model-agnostic, requiring no retraining and integrating seamlessly with diverse VLAs. To our knowledge, this is the first inference time concept based safety method for embodied systems, advancing both interpretability and safe deployment of VLA models.