# A Pattern Language for Resilient Visual Agents

arXiv: https://arxiv.org/abs/2604.28001 · HTML: https://mlsanity.com/paper/2604.28001/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in Multimodal and embodied agents, one of 16 directions in LLM agents and tool use.
- Builds on 10 of its 26 references that are in the map.
- First posted on arXiv 30 Apr 2026.

## Placed in

- LLM agents and tool use / [Multimodal and embodied agents](https://mlsanity.com/agents/multimodal-and-embodied-agents/)

## Builds on (most cited first, 10 of 10)

- [CogAgent: A Visual Language Model for GUI Agents](https://mlsanity.com/paper/2312.08914/) (2023)
- [UI-TARS: Pioneering Automated GUI Interaction with Native Agents](https://mlsanity.com/paper/2501.12326/) (2025)
- [OmniParser for Pure Vision Based GUI Agent](https://mlsanity.com/paper/2408.00203/) (2024)
- [ScreenAI: A Vision-Language Model for UI and Infographics Understanding](https://mlsanity.com/paper/2402.04615/) (2024)
- [Impact Of Explainable AI On Cognitive Load: Insights From An Empirical Study](https://mlsanity.com/paper/2304.08861/) (2023)
- [Fundamentals of Building Autonomous LLM Agents](https://mlsanity.com/paper/2510.09244/) (2025)
- [Towards Cognitive Bots: Architectural Research Challenges](https://mlsanity.com/paper/2305.17308/) (2023)
- [Modeling Adaptive Self-healing Systems](https://mlsanity.com/paper/2304.12773/) (2023)
- [User-Like Bots for Cognitive Automation: A Survey](https://mlsanity.com/paper/2311.12154/) (2023)
- [Affordance Representation and Recognition for Autonomous Agents](https://mlsanity.com/paper/2510.24459/) (2025)

## Built on by (most cited first, 1 of 1)

- [A Technical Taxonomy of LLM Agent Communication Protocols](https://mlsanity.com/paper/2606.19135/) (2026)

## Abstract (arXiv)

Integrating multimodal foundation models into enterprise ecosystems presents a fundamental software architecture challenge. Architects must balance competing quality attributes: the high latency and non-determinism of vision language action (VLA) models versus the strict determinism and real-time performance required by enterprise control loops. In this study, we propose an architectural pattern language for visual agents that separates fast, deterministic reflexes from slow, probabilistic supervision. It consists of four architectural design patterns: (1) Hybrid Affordance Integration, (2) Adaptive Visual Anchoring, (3) Visual Hierarchy Synthesis, and (4) Semantic Scene Graph.