mlsanityData to 14 Oct 2026

Skill Weaving: Efficient LLM Improvement via Modular Skillpacks

arXiv 2605.22205 ↗First posted 21 May 2026cs.AI cs.LGcited by 1

Not on the map. Why?

Abstract (arXiv)

Large language models increasingly require specialization across diverse domains, yet existing approaches struggle to balance multi-domain capacities with strict memory and inference constraints. In this work, we introduce SkillWeave, a modular improvement framework that enables LLMs to specialize under fixed memory budgets. SkillWeave partitions full capabilities of a general-purpose model into skillpacks -- lightweight, domain-specific delta modules -- that reorganize and refine the model's internal knowledge. For efficient deployment, SkillWeave integrates SkillZip to compress skillpacks into compact and inference-ready format, enabling strong multi-domain performance with low-latency execution. On multi-task and agentic benchmarks, a 9B SkillWeave model outperforms several baselines and even surpasses a 32B monolithic LLM, while achieving up to 4x speedup.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 45 of its 71 references that are in the map. First posted on arXiv 21 May 2026.

Builds on

45 of 71 references.

References · 45

  1. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  2. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  3. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  4. 2021Program Synthesis with Large Language Modelscited by 3,637
  5. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,919
  6. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,807
  7. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,386
  8. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,298
  9. 2024SimPO: Simple Preference Optimization with a Reference-Free Rewardcited by 963
  10. 2022LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scalecited by 906
  11. 2024Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorscited by 897
  12. 2023Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunchcited by 694
33 more
  1. 2023RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedbackcited by 625
  2. 2023Principle-Driven Self-Alignment of Language Models from Scratch with Minimal Human Supervisioncited by 318
  3. 2024SVD-LLM: Truncation-aware Singular Value Decomposition for Large Language Model Compressioncited by 243
  4. 2024Meta-Rewarding Language Models: Self-Improving Alignment with LLM-as-a-Meta-Judgecited by 204
  5. 2024Task Singular Vectors: Reducing Task Interference in Model Mergingcited by 147
  6. 2024Localizing Task Information for Improved Model Merging and Compressioncited by 131
  7. 2024Twin-Merging: Dynamic Integration of Modular Expertise in Model Mergingcited by 116
  8. 2024EMR-Merging: Tuning-Free High-Performance Model Mergingcited by 110
  9. 2023Task-Specific Skill Localization in Fine-tuned Language Modelscited by 105
  10. 2024Higher Layers Need More LoRA Expertscited by 102
  11. 2021Think you have Solved Direct-Answer Question Answering? Try ARC-DA, the Direct-Answer AI2 Reasoning Challengecited by 80
  12. 2024A Survey on Data Selection for LLM Instruction Tuningcited by 77
  13. 2024FuseChat: Knowledge Fusion of Chat Modelscited by 55
  14. 2024BitDelta: Your Fine-Tune May Only Be Worth One Bitcited by 39
  15. 2025MedEyes: Learning Dynamic Visual Focus for Medical Progressive Diagnosiscited by 32
  16. 2022Who Says Elephants Can't Run: Bringing Large Scale MoE Models into Cloud Scale Productioncited by 27
  17. 2025Jailbreak-R1: Exploring the Jailbreak Capabilities of LLMs via Reinforcement Learningcited by 27
  18. 2024Self-MoE: Towards Compositional Large Language Models with Self-Specialized Expertscited by 23
  19. 2024Knowledge Fusion By Evolving Weights of Language Modelscited by 16
  20. 2024Towards a Theoretical Understanding of Synthetic Data in LLM Post-Training: A Reverse-Bottleneck Perspectivecited by 15
  21. 2024Weighted-Reward Preference Optimization for Implicit Model Fusioncited by 15
  22. 2025Safety Alignment via Constrained Knowledge Unlearningcited by 14
  23. 2025Multi-objective Large Language Model Alignment with Hierarchical Expertscited by 13
  24. 2023Efficient Storage of Fine-Tuned Models via Low-Rank Approximation of Weight Residualscited by 12
  25. 2026MedSynapse-V: Bridging Visual Perception and Clinical Intuition via Latent Memory Evolutioncited by 12
  26. 2025Neural Parameter Search for Slimmer Fine-Tuned Models and Better Transfercited by 11
  27. 2025Self-Training Large Language Models with Confident Reasoningcited by 8
  28. 2025Knowledge Fusion of Large Language Models Via Modular SkillPackscited by 8
  29. 2024Self-Training Meets Consistency: Improving LLMs' Reasoning with Consistency-Driven Rationale Evaluationcited by 6
  30. 2025FuseChat-3.0: Preference Optimization Meets Heterogeneous Model Fusioncited by 6
  31. 2025Multi-Modality Expansion and Retention for LLMs through Parameter Merging and Decouplingcited by 5
  32. 2023End-to-end Rain Streak Removal with RAW Imagescited by 3
  33. 2026Dynamic Model Merging Made Slimcited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Inside the Skill Market: From Software Engineering Activities to Reusable Agent Skillscited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2026SkVM: Revisiting Language VM for Skills across Heterogenous LLMs and Harnesses
  2. 2026UniACE: A Unified Framework for Evaluating LLM Agentic Capabilities
  3. 2025Leveraging the true depth of LLMs
  4. 2025The Automated LLM Speedrunning Benchmark: Reproducing NanoGPT Improvements
  5. 2026Skills-Coach: A Self-Evolving Skill Optimizer via Training-Free GRPO
  6. 2025Bench360: Benchmarking Local LLM Inference from 360 Degrees
  7. 2025PLAY2PROMPT: Zero-shot Tool Instruction Optimization for LLM Agents via Tool Play
  8. 2024Small LLMs Are Weak Tool Learners: A Multi-LLM Agent