# Evolutionary Multi-Task Optimization for LLM-Guided Program Discovery

arXiv: https://arxiv.org/abs/2605.22613 · HTML: https://mlsanity.com/paper/2605.22613/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 44 references that are in the map.
- First posted on arXiv 21 May 2026.

## Builds on (most cited first, 20 of 20)

- [A Survey on Multi-Task Learning](https://mlsanity.com/paper/1707.08114/) (2017)
- [An Overview of Multi-Task Learning in Deep Neural Networks](https://mlsanity.com/paper/1706.05098/) (2017)
- [Promptbreeder: Self-Referential Self-Improvement Via Prompt Evolution](https://mlsanity.com/paper/2309.16797/) (2023)
- [GIFT-Eval: A Benchmark For General Time Series Forecasting Model Evaluation](https://mlsanity.com/paper/2410.10393/) (2024)
- [The Surprising Effectiveness of Test-Time Training for Few-Shot Learning](https://mlsanity.com/paper/2411.07279/) (2024)
- [LLMs and the Abstraction and Reasoning Corpus: Successes, Failures, and the Importance of Object-based Representations](https://mlsanity.com/paper/2305.18354/) (2023)
- [A Literature Survey of Benchmark Functions For Global Optimization Problems](https://mlsanity.com/paper/1308.4008/) (2013)
- [AdaEvolve: Adaptive LLM Driven Zeroth-Order Optimization](https://mlsanity.com/paper/2602.20133/) (2026)
- [Beyond Model Adaptation at Test Time: A Survey](https://mlsanity.com/paper/2411.03687/) (2024)
- [Test-Time Learning for Large Language Models](https://mlsanity.com/paper/2505.20633/) (2025)
- [DeltaEvolve: Accelerating Scientific Discovery through Momentum-Driven Evolution](https://mlsanity.com/paper/2602.02919/) (2026)
- [Warm Starting Bayesian Optimization](https://mlsanity.com/paper/1608.03585/) (2016)
- [Warm Starting CMA-ES for Hyperparameter Optimization](https://mlsanity.com/paper/2012.06932/) (2020)
- [Test-Time Training Provably Improves Transformers as In-context Learners](https://mlsanity.com/paper/2503.11842/) (2025)
- [PEFT A2Z: Parameter-Efficient Fine-Tuning Survey for Large Language and Vision Models](https://mlsanity.com/paper/2504.14117/) (2025)
- [Can Language Models Discover Scaling Laws?](https://mlsanity.com/paper/2507.21184/) (2025)
- [Identifying Task Groupings for Multi-Task Learning Using Pointwise V-Usable Information](https://mlsanity.com/paper/2410.12774/) (2024)
- [Multitask Evolution with Cartesian Genetic Programming](https://mlsanity.com/paper/1702.02217/) (2017)
- [NAACL2025 Tutorial: Adaptation of Large Language Models](https://mlsanity.com/paper/2504.03931/) (2025)
- [Seeding the Initial Population of Multi-Objective Evolutionary Algorithms: A Computational Study](https://mlsanity.com/paper/1412.0307/) (2014)

## Built on by (most cited first, 3 of 3)

- [DASyR-LLM: Domain-Aware Symbolic Regression with LLMs for Kinetic Model Discovery](https://mlsanity.com/paper/2608.05120/) (2026)
- [Multi-Task Evolution for Zero-Shot Cross-Problem Generalization using LLMs](https://mlsanity.com/paper/2610.03316/) (2026)
- [Discovered, Not Designed: Population Evolution for Collaborative and Compute-Intensive Model Discovery](https://mlsanity.com/paper/2610.05950/) (2026)

## Abstract (arXiv)

Recent LLM-guided evolutionary search methods have shown that iterative program mutation can discover strong algorithms, but they typically optimize each task independently, even when related tasks share reusable structure. We study evolutionary multitask optimization (EMO) for LLM-guided program discovery and propose EMO-STA (Shared-Then-Adapt), a two-stage framework that first evolves a shared archive of executable programs across a task family and then adapts selected shared candidates to each target task. Within EMO-STA, we explore multiple adaptation strategies, including warm-starting from the shared archive, adapting the best average shared program, and adapting the shared program that performs best on each target task. Across eight task families spanning continuous optimization, geometric construction, modeling, and algorithmic optimization, EMO-STA improves over single-task evolution under matched evolutionary-iteration budgets in most settings, with STA Best-Local providing the strongest in-distribution adaptation and STA Best-Shared yielding robust transfer to unseen tasks. Iteration-allocation experiments show that devoting a substantial fraction of the family-level iteration budget to shared evolution is effective, with roughly balanced shared and aggregate adaptation budgets often yielding the best performance. ARC and time-series feature-engineering case studies further suggest that shared evolution can mitigate overfitting in low-evidence settings by favoring programs that perform well across related tasks rather than exploiting brittle task-specific artifacts.