Why Solve It Twice? Hierarchical Accumulation of Skills for Transfer-Efficient ML Engineering
Not on the map. Why?
Abstract (arXiv)
ML engineering agents waste compute rediscovering known techniques because every competition is a cold start. We present HASTE, a hierarchical multi-agent system that organizes cross-competition knowledge into three scope tiers (global, domain, and competition-specific), each coupled to a matching agent level. An orchestrator coordinates domain specialists and promotes learning between tiers via LLM-driven abstraction. A controlled ablation provides evidence for scoped loading: holding a 159-skill inventory constant across 8 competitions, tiered loading achieves a 100% medal rate while flat loading reaches only 62.5%, the same medal rate as loading no skills, and consumes 2x the output tokens. On the full MLE-Bench Lite benchmark (22 Kaggle competitions), HASTE reaches a medal rate of 77.3% using Claude Sonnet 4.6 at 12h per competition; this is a single-seed campaign result, and multi-seed replication is the priority follow-up. In a cold-start run, the system begins with no accumulated skills. In warm-start runs, it reloads skills learned from earlier competitions, using only global and domain-level skills for transfer across competitions. Warm starts use 52% fewer refinement iterations, and the fraction of proposed changes kept by the agent rises from 42% at low inventory to 85% once 50+ skills are available. These results suggest that better knowledge organization can partly substitute for model strength and compute budget in ML-engineering agents.
Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 22 of its 35 references that are in the map. First posted on arXiv 29 Jun 2026.
Builds on
22 of 35 references.
References · 22
- 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,276
- 2023Voyager: An Open-Ended Embodied Agent with Large Language Modelscited by 2,191
- 2023ExpeL: LLM Agents Are Experiential Learnerscited by 876
- 2024MLE-bench: Evaluating Machine Learning Agents on Machine Learning Engineeringcited by 469
- 2023Cognitive Architectures for Language Agentscited by 429
- 2020AutoGluon-Tabular: Robust and Accurate AutoML for Structured Datacited by 383
- 2023Ghost in the Minecraft: Generally Capable Agents for Open-World Environments via Large Language Models with Text-based Knowledge and Memorycited by 289
- 2026SkillRL: Evolving Agents via Recursive Skill-Augmented Reinforcement Learningcited by 278
- 2025AIDE: AI-Driven Exploration in the Space of Codecited by 226
- 2020Auto-Sklearn 2.0: Hands-free AutoML via Meta-Learningcited by 174
- 2023Large Language Models for Automated Data Science: Introducing CAAFE for Context-Aware Automated Feature Engineeringcited by 127
- 2024DS-Agent: Automated Data Science by Empowering Large Language Models with Case-Based Reasoningcited by 125
10 more
- 2025On the Theoretical Limitations of Embedding-Based Retrievalcited by 104
- 2025AI Research Agents for Machine Learning: Search, Exploration, and Generalization in MLE-benchcited by 71
- 2023MLCopilot: Unleashing the Power of Large Language Models in Solving Machine Learning Taskscited by 66
- 2024Large Language Model Agent for Hyper-Parameter Optimizationcited by 64
- 2025MLE-STAR: Machine Learning Engineering Agent via Search and Targeted Refinementcited by 58
- 2024VLM Agents Generate Their Own Memories: Distilling Experience into Embodied Programs of Thoughtcited by 55
- 2024Kolb-Based Experiential Learning for Generalist Agents with Human-Level Kaggle Data Science Performancecited by 25
- 2025MLZero: A Multi-Agent System for End-to-end Machine Learning Automationcited by 24
- 2025LoongFlow: Directed Evolutionary Search via a Cognitive Plan-Execute-Summarize Paradigmcited by 22
- 2026Toward a Theory of Hierarchical Memory for Language Agentscited by 5
Built on by
Citing papers per year: 2026: 3.
3 papers.