mlsanityData to 14 Oct 2026

Penny Wise, Pixel Foolish: Bypassing Price Constraints in Multimodal Agents via Visual Adversarial Perturbations

arXiv 2604.16515 ↗First posted 15 Apr 2026cs.CV cs.CR cs.LGcited by 3

On the map: Multimodal LLMs / vision-language models → Chart understanding and anomaly detection

Abstract (arXiv)

The rapid proliferation of Multimodal Large Language Models (MLLMs) has enabled mobile agents to execute high-stakes financial transactions, but their adversarial robustness remains underexplored. We identify Visual Dominance Hallucination (VDH), where imperceptible visual cues can override textual price evidence in screenshot-based, price-constrained settings and lead agents to irrational decisions. We propose PriceBlind, a stealthy white-box adversarial attack framework for controlled screenshot-based evaluation. PriceBlind exploits the modality gap in CLIP-based encoders via a Semantic-Decoupling Loss that aligns the image embedding with low-cost, value-associated anchors while preserving pixel-level fidelity. On E-ShopBench, PriceBlind achieves around 80% ASR in white-box evaluation; under a simplified single-turn coordinate-selection protocol, Ensemble-DI-FGSM transfers with roughly 35-41% ASR across GPT-4o, Gemini-1.5-Pro, and Claude-3.5-Sonnet. We also show that robust encoders and Verify-then-Act defenses reduce ASR substantially, though with some clean-accuracy trade-off.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Placed in Chart understanding and anomaly detection, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 17 of its 35 references that are in the map. First posted on arXiv 15 Apr 2026.

Builds on

17 of 35 references.

References · 17

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2014Explaining and Harnessing Adversarial Examplescited by 7,364
  3. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,415
  4. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,109
  5. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,224
  6. 2020Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attackscited by 1,451
  7. 2018Improving Transferability of Adversarial Examples with Input Diversitycited by 599
  8. 2023AppAgent: Multimodal Agents as Smartphone Userscited by 415
  9. 2023AutoDroid: LLM-powered Task Automation in Androidcited by 184
  10. 2020A Self-supervised Approach for Adversarial Robustnesscited by 180
  11. 2024OmniACT: A Dataset and Benchmark for Enabling Multimodal Generalist Autonomous Agents for Desktop and Webcited by 157
  12. 2024CoCo-Agent: A Comprehensive Cognitive MLLM Agent for Smartphone GUI Automationcited by 70
5 more
  1. 2023Adversarial Illusions in Multi-Modal Embeddingscited by 29
  2. 2025Visual Contextual Attack: Jailbreaking MLLMs with Image-Driven Context Injectioncited by 25
  3. 2022FARE: Provably Fair Representation Learning with Practical Certificatescited by 21
  4. 2025GhostEI-Bench: Do Mobile Agents Resilience to Environmental Injection in Dynamic On-Device Environments?cited by 12
  5. 2025AdPO: Enhancing the Adversarial Robustness of Large Vision-Language Models with Preference Optimizationcited by 8

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026JECA^2: Judgment-Explanation Consistent Adversarial Attack against Forensic Vision-Language Modelscited by 0
  2. 2026VisCritic: Visual State Comparison as Process Reward for GUI Agentscited by 0
  3. 2026SAVOR: Self-Aware Visual Grounding via Confidence-Calibrated Reinforcement Learning for Multimodal Hallucination Mitigationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Market Signal Injection: Adversarial Context Manipulation of LLM Pricing Agents
  2. 2026Contagion on the Trading Floor: How Adversarial Signals Spread in Multi-Agent Trading Systems
  3. 2025PyFi: Toward Pyramid-like Financial Image Understanding for VLMs via Adversarial Agents
  4. 2024Iteratively Prompting Multimodal LLMs to Reproduce Natural and AI-Generated Images
  5. 2026Your Agent Says Yes: Interpreting Adversarial Market Behavior Beyond Individual Transactions
  6. 2026Forced Deferral: Manipulating Routing Decisions in Multimodal LLM Cascades
  7. 2026FinMTM: A Multi-Turn Multimodal Benchmark for Financial Reasoning and Agent Evaluation
  8. 2026FinRT: Distilling Adaptive Red-Teaming Strategies into Reusable Adversarial Generators in Consumer Finance