Agent Skills Should Go Beyond Text: The Case for Visual Skills
Not on the map. Why?
Abstract (arXiv)
Reusable skills are a key mechanism for extending agent capabilities, allowing agents to accumulate experience and solve increasingly complex tasks. Yet most existing skill-learning methods store reusable experience as text-only assets, such as instructions, reasoning traces, or summarized trajectories. We argue that this text-only paradigm creates a fundamental bottleneck for visual-centric tasks, where reusable knowledge often depends on spatial layout, visual grounding, fine-grained appearance, and localized state changes. To address this limitation, we propose NAME, a multimodal skill paradigm that combines declarative textual logic with explicit visual support. We distinguish three reusable forms: static priors for stable spatial conventions, dynamic priors for in-situ visual working memory, and interleaved visual skills that bind ordered text steps to the source frames, screenshots, or page regions that justify them. Rather than only describing what to do, visual skills also encode where to look, how to inspect, and how to verify visual outcomes. To scale visual-skill construction, we introduce SYSTEM, an automatic system that converts agent experience into reusable multimodal skills by preserving textual reasoning, spatial references, visual boundaries, and interaction patterns from task trajectories. Experiments on GUI and other visual-centric tasks show that visual skills consistently outperform text-only skills, particularly when success requires spatial correspondence, visual evidence, and state-aware interaction. These results support our central position: reusable agent skills should go beyond text and become multimodal assets for future multimodal agents.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 31 of its 45 references that are in the map. First posted on arXiv 31 May 2026.
Builds on
31 of 45 references.
References · 31
- 2023GPT-4 Technical Reportcited by 18,132
- 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
- 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,072
- 2023Qwen Technical Reportcited by 3,273
- 2023WebArena: A Realistic Web Environment for Building Autonomous Agentscited by 1,908
- 2023AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversationcited by 1,906
- 2023Mind2Web: Towards a Generalist Agent for the Webcited by 1,210
- 2024PaliGemma: A versatile 3B VLM for transfercited by 679
- 2024SeeClick: Harnessing GUI Grounding for Advanced Visual GUI Agentscited by 553
- 2022Winoground: Probing Vision and Language Models for Visio-Linguistic Compositionalitycited by 523
- 2022Visual Programming: Compositional visual reasoning without trainingcited by 507
- 2024OS-ATLAS: A Foundation Action Model for Generalist GUI Agentscited by 370
19 more
- 2023SugarCrepe: Fixing Hackable Benchmarks for Vision-Language Compositionalitycited by 238
- 2023Teaching CLIP to Count to Tencited by 194
- 2025Mobile-Agent-v3: Fundamental Agents for GUI Automationcited by 183
- 2024Design2Code: Benchmarking Multimodal Code Generation for Automated Front-End Engineeringcited by 146
- 2024Unlocking the conversion of Web Screenshots into HTML Code with the WebSight Datasetcited by 126
- 2025UI-Venus Technical Report: Building High-performance UI Agents with RFTcited by 90
- 2025MAI-UI Technical Report: Real-World Centric Foundation GUI Agentscited by 88
- 2023PosterLayout: A New Benchmark and Approach for Content-aware Visual-Textual Presentation Layoutcited by 60
- 2024AgentStudio: A Toolkit for Building General Virtual Agentscited by 45
- 2024V2Xum-LLM: Cross-Modal Video Summarization with Temporal Prompt Instruction Tuningcited by 45
- 2024MMCOMPOSITION: Revisiting the Compositionality of Pre-trained Vision-Language Modelscited by 29
- 2025Latent Chain-of-Thought for Visual Reasoningcited by 27
- 2025Building a Foundational Guardrail for General Agentic Systems via Synthetic Datacited by 26
- 2024FINEMATCH: Aspect-based Fine-grained Image and Text Mismatch Detection and Correctioncited by 16
- 2025MMIG-Bench: Towards Comprehensive and Explainable Evaluation of Multi-Modal Image Generation Modelscited by 16
- 2025MIRA: Multimodal Iterative Reasoning Agent for Image Editingcited by 15
- 2026JobBench: Aligning Agent Work With Human Willcited by 10
- 2026MementoGUI: Learning Agentic Multimodal Memory Control for Long-Horizon GUI Agentscited by 9
- 2026Aurora: Unified Video Editing with a Tool-Using Agentcited by 7
Built on by
Citing papers per year: 2026: 1.
1 paper.
Similar, not cited
Close in wording, no citation either way
- 2026MMSkills: Towards Multimodal Skills for General Visual Agents
- 2026V-Gym: Enhancing Agentic Visual Reasoning via Skill-Data Co-Evolution
- 2026VISUALSKILL: Multimodal Skills for Computer-Use Agents
- 2026From Skill Text to Skill Structure: The Scheduling-Structural-Logical Representation for Agent Skills
- 2026SkillAlign: Aligning Skill Interfaces for LLM-based Agents
- 2026XSkill: Continual Learning from Experience and Skills in Multimodal Agents
- 2026LatentSkill: From In-Context Textual Skills to In-Weight Latent Skills for LLM Agents
- 2026Skill-Pro: Learning Reusable Skills from Experience via Non-Parametric PPO for LLM Agents