# Model Context Protocol for Vision Systems: Audit, Security, and Protocol Extensions

arXiv: https://arxiv.org/abs/2509.22814 · HTML: https://mlsanity.com/paper/2509.22814/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in Agent threats, guardrails and governance, one of 16 directions in LLM agents and tool use.
- Builds on 17 of its 36 references that are in the map.
- First posted on arXiv 26 Sep 2025.

## Placed in

- LLM agents and tool use / [Agent threats, guardrails and governance](https://mlsanity.com/agents/agent-threats-guardrails-and-governance/)

## Builds on (most cited first, 17 of 17)

- [Toolformer: Language Models Can Teach Themselves to Use Tools](https://mlsanity.com/paper/2302.04761/) (2023)
- [A Survey on Large Language Model based Autonomous Agents](https://mlsanity.com/paper/2308.11432/) (2023)
- [Chain of Agents: Large Language Models Collaborating on Long-Context Tasks](https://mlsanity.com/paper/2406.02818/) (2024)
- [LLaVA-Plus: Learning to Use Tools for Creating Multimodal Agents](https://mlsanity.com/paper/2311.05437/) (2023)
- [MultiBench: Multiscale Benchmarks for Multimodal Representation Learning](https://mlsanity.com/paper/2107.07502/) (2021)
- [Magma: A Foundation Model for Multimodal AI Agents](https://mlsanity.com/paper/2502.13130/) (2025)
- [Seeing biodiversity: perspectives in machine learning for wildlife conservation](https://mlsanity.com/paper/2110.12951/) (2021)
- [Alita: Generalist Agent Enabling Scalable Agentic Reasoning with Minimal Predefinition and Maximal Self-Evolution](https://mlsanity.com/paper/2505.20286/) (2025)
- [We Should Identify and Mitigate Third-Party Safety Risks in MCP-Powered Agent Systems](https://mlsanity.com/paper/2506.13666/) (2025)
- [Enhancing Multimodal Compositional Reasoning of Visual Language Models with Generative Negative Mining](https://mlsanity.com/paper/2311.03964/) (2023)
- [Iterative Tool Usage Exploration for Multimodal Agents via Step-wise Preference Tuning](https://mlsanity.com/paper/2504.21561/) (2025)
- [We Urgently Need Privilege Management in MCP: A Measurement of API Usage in MCP Ecosystems](https://mlsanity.com/paper/2507.06250/) (2025)
- [VISION: A Modular AI Assistant for Natural Human-Instrument Interaction at Scientific User Facilities](https://mlsanity.com/paper/2412.18161/) (2024)
- [A Large-Scale Evolvable Dataset for Model Context Protocol Ecosystem and Security Analysis](https://mlsanity.com/paper/2506.23474/) (2025)
- [Prompting Large Vision-Language Models for Compositional Reasoning](https://mlsanity.com/paper/2401.11337/) (2024)
- [Enhancing Clinical Decision Support and EHR Insights through LLMs and the Model Context Protocol: An Open-Source MCP-FHIR Framework](https://mlsanity.com/paper/2506.13800/) (2025)
- [Enhancing Compositional Generalization via Compositional Feature Alignment](https://mlsanity.com/paper/2402.02851/) (2024)

## Built on by (most cited first, 1 of 1)

- [Model Context Protocol (MCP) Tool Descriptions Are Smelly! Towards Improving AI Agent Efficiency with Augmented MCP Tool Descriptions](https://mlsanity.com/paper/2602.14878/) (2026)

## Abstract (arXiv)

The Model Context Protocol (MCP) defines a schema bound execution model for agent-tool interaction, enabling modular computer vision workflows without retraining. To our knowledge, this is the first protocol level, deployment scale audit of MCP in vision systems, identifying systemic weaknesses in schema semantics, interoperability, and runtime coordination. We analyze 91 publicly registered vision centric MCP servers, annotated along nine dimensions of compositional fidelity, and develop an executable benchmark with validators to detect and categorize protocol violations. The audit reveals high prevalence of schema format divergence, missing runtime schema validation, undeclared coordinate conventions, and reliance on untracked bridging scripts. Validator based testing quantifies these failures, with schema format checks flagging misalignments in 78.0 percent of systems, coordinate convention checks detecting spatial reference errors in 24.6 percent, and memory scope checks issuing an average of 33.8 warnings per 100 executions. Security probes show that dynamic and multi agent workflows exhibit elevated risks of privilege escalation and untyped tool connections. The proposed benchmark and validator suite, implemented in a controlled testbed and to be released on GitHub, establishes a reproducible framework for measuring and improving the reliability and security of compositional vision workflows.