# Why Solve It Twice? Hierarchical Accumulation of Skills for Transfer-Efficient ML Engineering

arXiv: https://arxiv.org/abs/2606.30911 · HTML: https://mlsanity.com/paper/2606.30911/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 35 references that are in the map.
- First posted on arXiv 29 Jun 2026.

## Builds on (most cited first, 22 of 22)

- [Reflexion: Language Agents with Verbal Reinforcement Learning](https://mlsanity.com/paper/2303.11366/) (2023)
- [Voyager: An Open-Ended Embodied Agent with Large Language Models](https://mlsanity.com/paper/2305.16291/) (2023)
- [ExpeL: LLM Agents Are Experiential Learners](https://mlsanity.com/paper/2308.10144/) (2023)
- [MLE-bench: Evaluating Machine Learning Agents on Machine Learning Engineering](https://mlsanity.com/paper/2410.07095/) (2024)
- [Cognitive Architectures for Language Agents](https://mlsanity.com/paper/2309.02427/) (2023)
- [AutoGluon-Tabular: Robust and Accurate AutoML for Structured Data](https://mlsanity.com/paper/2003.06505/) (2020)
- [Ghost in the Minecraft: Generally Capable Agents for Open-World Environments via Large Language Models with Text-based Knowledge and Memory](https://mlsanity.com/paper/2305.17144/) (2023)
- [SkillRL: Evolving Agents via Recursive Skill-Augmented Reinforcement Learning](https://mlsanity.com/paper/2602.08234/) (2026)
- [AIDE: AI-Driven Exploration in the Space of Code](https://mlsanity.com/paper/2502.13138/) (2025)
- [Auto-Sklearn 2.0: Hands-free AutoML via Meta-Learning](https://mlsanity.com/paper/2007.04074/) (2020)
- [Large Language Models for Automated Data Science: Introducing CAAFE for Context-Aware Automated Feature Engineering](https://mlsanity.com/paper/2305.03403/) (2023)
- [DS-Agent: Automated Data Science by Empowering Large Language Models with Case-Based Reasoning](https://mlsanity.com/paper/2402.17453/) (2024)
- [On the Theoretical Limitations of Embedding-Based Retrieval](https://mlsanity.com/paper/2508.21038/) (2025)
- [AI Research Agents for Machine Learning: Search, Exploration, and Generalization in MLE-bench](https://mlsanity.com/paper/2507.02554/) (2025)
- [MLCopilot: Unleashing the Power of Large Language Models in Solving Machine Learning Tasks](https://mlsanity.com/paper/2304.14979/) (2023)
- [Large Language Model Agent for Hyper-Parameter Optimization](https://mlsanity.com/paper/2402.01881/) (2024)
- [MLE-STAR: Machine Learning Engineering Agent via Search and Targeted Refinement](https://mlsanity.com/paper/2506.15692/) (2025)
- [VLM Agents Generate Their Own Memories: Distilling Experience into Embodied Programs of Thought](https://mlsanity.com/paper/2406.14596/) (2024)
- [Kolb-Based Experiential Learning for Generalist Agents with Human-Level Kaggle Data Science Performance](https://mlsanity.com/paper/2411.03562/) (2024)
- [MLZero: A Multi-Agent System for End-to-end Machine Learning Automation](https://mlsanity.com/paper/2505.13941/) (2025)
- [LoongFlow: Directed Evolutionary Search via a Cognitive Plan-Execute-Summarize Paradigm](https://mlsanity.com/paper/2512.24077/) (2025)
- [Toward a Theory of Hierarchical Memory for Language Agents](https://mlsanity.com/paper/2603.21564/) (2026)

## Built on by (most cited first, 3 of 3)

- [Frontis-MA1: Training an AI4AI Model towards Recursive Self-Improvement in Machine Learning Engineering](https://mlsanity.com/paper/2607.28568/) (2026)
- [Toward Personal Intelligence Through Cooperative Observation](https://mlsanity.com/paper/2608.17128/) (2026)
- [Strategy Accumulation and Guided Execution for Automated LLM Fine-Tuning](https://mlsanity.com/paper/2609.22257/) (2026)

## Abstract (arXiv)

ML engineering agents waste compute rediscovering known techniques because every competition is a cold start. We present HASTE, a hierarchical multi-agent system that organizes cross-competition knowledge into three scope tiers (global, domain, and competition-specific), each coupled to a matching agent level. An orchestrator coordinates domain specialists and promotes learning between tiers via LLM-driven abstraction. A controlled ablation provides evidence for scoped loading: holding a 159-skill inventory constant across 8 competitions, tiered loading achieves a 100% medal rate while flat loading reaches only 62.5%, the same medal rate as loading no skills, and consumes 2x the output tokens. On the full MLE-Bench Lite benchmark (22 Kaggle competitions), HASTE reaches a medal rate of 77.3% using Claude Sonnet 4.6 at 12h per competition; this is a single-seed campaign result, and multi-seed replication is the priority follow-up. In a cold-start run, the system begins with no accumulated skills. In warm-start runs, it reloads skills learned from earlier competitions, using only global and domain-level skills for transfer across competitions. Warm starts use 52% fewer refinement iterations, and the fraction of proposed changes kept by the agent rises from 42% at low inventory to 85% once 50+ skills are available. These results suggest that better knowledge organization can partly substitute for model strength and compute budget in ML-engineering agents.