# Exploring the Robustness of Decision-Level Through Adversarial Attacks on LLM-Based Embodied Models

arXiv: https://arxiv.org/abs/2405.19802 · HTML: https://mlsanity.com/paper/2405.19802/

## Key facts

- Cited by 33 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 30 May 2024.

## Built on by (most cited first, 30 of 33)

- [SafeAgentBench: A Benchmark for Safe Task Planning of Embodied LLM Agents](https://mlsanity.com/paper/2412.13178/) (2024)
- [BadRobot: Jailbreaking Embodied LLM Agents in the Physical World](https://mlsanity.com/paper/2407.20242/) (2024)
- [IS-Bench: Evaluating Interactive Safety of VLM-Driven Embodied Agents in Daily Household Tasks](https://mlsanity.com/paper/2506.16402/) (2025)
- [AGENTSAFE: Benchmarking the Safety of Embodied Agents on Hazardous Instructions](https://mlsanity.com/paper/2506.14697/) (2025)
- [MLA-Trust: Benchmarking Trustworthiness of Multimodal LLM Agents in GUI Environments](https://mlsanity.com/paper/2506.01616/) (2025)
- [ADVEDM:Fine-grained Adversarial Attack against VLM-based Embodied Agents](https://mlsanity.com/paper/2509.16645/) (2025)
- [SafeMind: Benchmarking and Mitigating Safety Risks in Embodied LLM Agents](https://mlsanity.com/paper/2509.25885/) (2025)
- [Robot Collapse: Supply Chain Backdoor Attacks Against VLM-based Robotic Manipulation](https://mlsanity.com/paper/2411.11683/) (2024)
- [Easier Said Than Done: Unpacking Intent-Behavior Gap in Jailbreaking LLM-Based Robots](https://mlsanity.com/paper/2412.16633/) (2024)
- [Safety of Embodied Navigation: A Survey](https://mlsanity.com/paper/2508.05855/) (2025)
- [Generalizability of Large Language Model-Based Agents: A Comprehensive Survey](https://mlsanity.com/paper/2509.16330/) (2025)
- [State Backdoor: Towards Stealthy Real-world Poisoning Attack on Vision-Language-Action Model in State Space](https://mlsanity.com/paper/2601.04266/) (2026)
- [Toward Secure LLM Agents: Threat Surfaces, Attacks, Defenses, and Evaluation](https://mlsanity.com/paper/2606.10749/) (2026)
- [What Breaks Embodied AI Security:LLM Vulnerabilities, CPS Flaws,or Something Else?](https://mlsanity.com/paper/2602.17345/) (2026)
- [From Pixels to Tokens: Revisiting Object Hallucinations in Large Vision-Language Models](https://mlsanity.com/paper/2410.06795/) (2024)
- [HomeSafeBench: Benchmarking Embodied Vision-Language Models in Free-Exploration Home Safety Inspection](https://mlsanity.com/paper/2509.23690/) (2025)
- [Benchmarking and Improving LLM Robustness for Personalized Generation](https://mlsanity.com/paper/2509.19358/) (2025)
- [Partially Observable Adversarial Patch Attacks on Vision-Language-Action Models in Robotics](https://mlsanity.com/paper/2606.03556/) (2026)
- [ANNIE: Be Careful of Your Robots](https://mlsanity.com/paper/2509.03383/) (2025)
- [Disrupting Vision-Language Model-Driven Navigation Services via Adversarial Object Fusion](https://mlsanity.com/paper/2505.23266/) (2025)
- [RoboInspector: Unveiling the Unreliability of Policy Code for LLM-enabled Robotic Manipulation](https://mlsanity.com/paper/2508.21378/) (2025)
- [MuTRAP: Multi-trigger Trojans Attacking Robot Task Planning Systems](https://mlsanity.com/paper/2504.17070/) (2025)
- [Unfettered Forceful Skill Acquisition with Physical Reasoning and Coordinate Frame Labeling](https://mlsanity.com/paper/2505.09731/) (2025)
- [Exploring the Secondary Risks of Large Language Models](https://mlsanity.com/paper/2506.12382/) (2025)
- [Rethinking Robust Adversarial Concept Erasure in Diffusion Models](https://mlsanity.com/paper/2510.27285/) (2025)
- [VLGOR: Visual-Language Knowledge Guided Offline Reinforcement Learning for Generalizable Agents](https://mlsanity.com/paper/2603.22892/) (2026)
- [On the Dual-Use Dilemma in Physical Reasoning and Force](https://mlsanity.com/paper/2505.18792/) (2025)
- [Beyond Model Jailbreak: Systematic Dissection of the "Ten DeadlySins" in Embodied Intelligence](https://mlsanity.com/paper/2512.06387/) (2025)
- [LCO: LLM-based Constraint Optimization for Safer Agentic LLMs in Real-world Tasks](https://mlsanity.com/paper/2605.27375/) (2026)
- [AutoSpec: Safety Rule Evolution for LLM Agents via Inductive Logic Programming](https://mlsanity.com/paper/2606.24245/) (2026)

## Abstract (arXiv)

Embodied intelligence empowers agents with a profound sense of perception, enabling them to respond in a manner closely aligned with real-world situations. Large Language Models (LLMs) delve into language instructions with depth, serving a crucial role in generating plans for intricate tasks. Thus, LLM-based embodied models further enhance the agent's capacity to comprehend and process information. However, this amalgamation also ushers in new challenges in the pursuit of heightened intelligence. Specifically, attackers can manipulate LLMs to produce irrelevant or even malicious outputs by altering their prompts. Confronted with this challenge, we observe a notable absence of multi-modal datasets essential for comprehensively evaluating the robustness of LLM-based embodied models. Consequently, we construct the Embodied Intelligent Robot Attack Dataset (EIRAD), tailored specifically for robustness evaluation. Additionally, two attack strategies are devised, including untargeted attacks and targeted attacks, to effectively simulate a range of diverse attack scenarios. At the same time, during the attack process, to more accurately ascertain whether our method is successful in attacking the LLM-based embodied model, we devise a new attack success evaluation method utilizing the BLIP2 model. Recognizing the time and cost-intensive nature of the GCG algorithm in attacks, we devise a scheme for prompt suffix initialization based on various target tasks, thus expediting the convergence process. Experimental results demonstrate that our method exhibits a superior attack success rate when targeting LLM-based embodied models, indicating a lower level of decision-level robustness in these models.