RESOURCE2SKILL: Distilling Executable Agent Skills from Human-Created Multimodal Resources
Not on the map. Why?
Abstract (arXiv)
Skills are a useful abstraction for software agents, turning human and agent experience into reusable procedural knowledge. Yet existing skill libraries are mostly hand-written, text-centric, or derived from agent traces, leaving tutorial videos and other multimodal human resources largely underused. We present RESOURCE2SKILL, a framework that distills multimodal resources, including tutorial videos, repositories, articles, and reference artifacts, into executable skills for software agents. RESOURCE2SKILL organizes these skills as a hierarchical multimodal Skill Wiki, where each entry combines structured text, code, visual examples, metadata, and provenance. This design preserves complementary signals from different resources: videos capture temporal operations and visual effects, code captures executable tool patterns, and articles or artifacts provide conceptual and stylistic grounding. At inference time, agents retrieve and compose relevant skills from the wiki; when coverage is insufficient, the same construction operator can acquire new skills online. Across seven practical authoring domains, RESOURCE2SKILL improves average overall score by +11.9 percentage points over no-skill agents and outperforms strong harness baselines in 26 of 28 main-aggregate model-domain cells. Ablations confirm the value of multimodal skill format, hierarchical organization, source diversity, selection strategy, and online acquisition.
Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 37 of its 40 references that are in the map. First posted on arXiv 28 Jun 2026.
Builds on
37 of 40 references.
References · 37
- 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
- 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
- 2021Evaluating Large Language Models Trained on Codecited by 8,320
- 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,276
- 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,182
- 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,824
- 2023Generative Agents: Interactive Simulacra of Human Behaviorcited by 3,568
- 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,442
- 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,373
- 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,348
- 2023Voyager: An Open-Ended Embodied Agent with Large Language Modelscited by 2,191
- 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,141
25 more
- 2023WebArena: A Realistic Web Environment for Building Autonomous Agentscited by 2,006
- 2023ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIscited by 1,947
- 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,745
- 2024SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineeringcited by 1,737
- 2021Ego4D: Around the World in 3,000 Hours of Egocentric Videocited by 1,556
- 2023MemGPT: Towards LLMs as Operating Systemscited by 1,389
- 2023Gorilla: Large Language Model Connected with Massive APIscited by 1,332
- 2021Improving language models by retrieving from trillions of tokenscited by 1,293
- 2023AgentBench: Evaluating LLMs as Agentscited by 1,267
- 2022WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agentscited by 1,251
- 2023Mind2Web: Towards a Generalist Agent for the Webcited by 1,242
- 2022Code as Policies: Language Model Programs for Embodied Controlcited by 1,210
- 2023HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Facecited by 1,190
- 2019HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clipscited by 950
- 2019CodeSearchNet Challenge: Evaluating the State of Semantic Code Searchcited by 654
- 2022PAL: Program-aided Language Modelscited by 652
- 2022MineDojo: Building Open-Ended Embodied Agents with Internet-Scale Knowledgecited by 478
- 2024Agent Workflow Memorycited by 383
- 2019COIN: A Large-scale Dataset for Comprehensive Instructional Video Analysiscited by 318
- 2019Cross-task weakly supervised learning from instructional videoscited by 199
- 2022MRKL Systems: A modular, neuro-symbolic architecture that combines large language models, external knowledge sources and discrete reasoningcited by 156
- 2026Agent Skills for Large Language Models: Architecture, Acquisition, Security, and the Path Forwardcited by 113
- 2025Inducing Programmatic Skills for Agentic Taskscited by 111
- 2026SKILLFOUNDRY: Building Self-Evolving Agent Skill Libraries from Heterogeneous Scientific Resourcescited by 41
- 2026SkillFlow:Benchmarking Lifelong Skill Discovery and Evolution for Autonomous Agentscited by 25
Built on by
Citing papers per year: 2026: 2.
2 papers.