mlsanityData to 7 Oct 2026

Agent Skills Should Go Beyond Text: The Case for Visual Skills

arXiv 2606.01414 ↗First posted 31 May 2026cs.CVcited by 1

Not on the map. Why?

Abstract (arXiv)

Reusable skills are a key mechanism for extending agent capabilities, allowing agents to accumulate experience and solve increasingly complex tasks. Yet most existing skill-learning methods store reusable experience as text-only assets, such as instructions, reasoning traces, or summarized trajectories. We argue that this text-only paradigm creates a fundamental bottleneck for visual-centric tasks, where reusable knowledge often depends on spatial layout, visual grounding, fine-grained appearance, and localized state changes. To address this limitation, we propose NAME, a multimodal skill paradigm that combines declarative textual logic with explicit visual support. We distinguish three reusable forms: static priors for stable spatial conventions, dynamic priors for in-situ visual working memory, and interleaved visual skills that bind ordered text steps to the source frames, screenshots, or page regions that justify them. Rather than only describing what to do, visual skills also encode where to look, how to inspect, and how to verify visual outcomes. To scale visual-skill construction, we introduce SYSTEM, an automatic system that converts agent experience into reusable multimodal skills by preserving textual reasoning, spatial references, visual boundaries, and interaction patterns from task trajectories. Experiments on GUI and other visual-centric tasks show that visual skills consistently outperform text-only skills, particularly when success requires spatial correspondence, visual evidence, and state-aware interaction. These results support our central position: reusable agent skills should go beyond text and become multimodal assets for future multimodal agents.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 31 of its 45 references that are in the map. First posted on arXiv 31 May 2026.

Builds on

31 of 45 references.

References · 31

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  3. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,072
  4. 2023Qwen Technical Reportcited by 3,273
  5. 2023WebArena: A Realistic Web Environment for Building Autonomous Agentscited by 1,908
  6. 2023AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversationcited by 1,906
  7. 2023Mind2Web: Towards a Generalist Agent for the Webcited by 1,210
  8. 2024PaliGemma: A versatile 3B VLM for transfercited by 679
  9. 2024SeeClick: Harnessing GUI Grounding for Advanced Visual GUI Agentscited by 553
  10. 2022Winoground: Probing Vision and Language Models for Visio-Linguistic Compositionalitycited by 523
  11. 2022Visual Programming: Compositional visual reasoning without trainingcited by 507
  12. 2024OS-ATLAS: A Foundation Action Model for Generalist GUI Agentscited by 370
19 more
  1. 2023SugarCrepe: Fixing Hackable Benchmarks for Vision-Language Compositionalitycited by 238
  2. 2023Teaching CLIP to Count to Tencited by 194
  3. 2025Mobile-Agent-v3: Fundamental Agents for GUI Automationcited by 183
  4. 2024Design2Code: Benchmarking Multimodal Code Generation for Automated Front-End Engineeringcited by 146
  5. 2024Unlocking the conversion of Web Screenshots into HTML Code with the WebSight Datasetcited by 126
  6. 2025UI-Venus Technical Report: Building High-performance UI Agents with RFTcited by 90
  7. 2025MAI-UI Technical Report: Real-World Centric Foundation GUI Agentscited by 88
  8. 2023PosterLayout: A New Benchmark and Approach for Content-aware Visual-Textual Presentation Layoutcited by 60
  9. 2024AgentStudio: A Toolkit for Building General Virtual Agentscited by 45
  10. 2024V2Xum-LLM: Cross-Modal Video Summarization with Temporal Prompt Instruction Tuningcited by 45
  11. 2024MMCOMPOSITION: Revisiting the Compositionality of Pre-trained Vision-Language Modelscited by 29
  12. 2025Latent Chain-of-Thought for Visual Reasoningcited by 27
  13. 2025Building a Foundational Guardrail for General Agentic Systems via Synthetic Datacited by 26
  14. 2024FINEMATCH: Aspect-based Fine-grained Image and Text Mismatch Detection and Correctioncited by 16
  15. 2025MMIG-Bench: Towards Comprehensive and Explainable Evaluation of Multi-Modal Image Generation Modelscited by 16
  16. 2025MIRA: Multimodal Iterative Reasoning Agent for Image Editingcited by 15
  17. 2026JobBench: Aligning Agent Work With Human Willcited by 10
  18. 2026MementoGUI: Learning Agentic Multimodal Memory Control for Long-Horizon GUI Agentscited by 9
  19. 2026Aurora: Unified Video Editing with a Tool-Using Agentcited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.

Similar, not cited

Close in wording, no citation either way

  1. 2026MMSkills: Towards Multimodal Skills for General Visual Agents
  2. 2026V-Gym: Enhancing Agentic Visual Reasoning via Skill-Data Co-Evolution
  3. 2026VISUALSKILL: Multimodal Skills for Computer-Use Agents
  4. 2026From Skill Text to Skill Structure: The Scheduling-Structural-Logical Representation for Agent Skills
  5. 2026SkillAlign: Aligning Skill Interfaces for LLM-based Agents
  6. 2026XSkill: Continual Learning from Experience and Skills in Multimodal Agents
  7. 2026LatentSkill: From In-Context Textual Skills to In-Weight Latent Skills for LLM Agents
  8. 2026Skill-Pro: Learning Reusable Skills from Experience via Non-Parametric PPO for LLM Agents