# Evolution Fine-Tuning: Learning to Discover Across 371 Optimization Tasks

arXiv: https://arxiv.org/abs/2606.29082 · HTML: https://mlsanity.com/paper/2606.29082/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 46 of its 100 references that are in the map.
- First posted on arXiv 27 Jun 2026.

## Builds on (most cited first, 30 of 46)

- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Models](https://mlsanity.com/paper/2403.13372/) (2024)
- [AlphaEvolve: A coding agent for scientific and algorithmic discovery](https://mlsanity.com/paper/2506.13131/) (2025)
- [The Flan Collection: Designing Data and Methods for Effective Instruction Tuning](https://mlsanity.com/paper/2301.13688/) (2023)
- [GEPA: Reflective Prompt Evolution Can Outperform Reinforcement Learning](https://mlsanity.com/paper/2507.19457/) (2025)
- [Darwin Godel Machine: Open-Ended Evolution of Self-Improving Agents](https://mlsanity.com/paper/2505.22954/) (2025)
- [Meta-Harness: End-to-End Optimization of Model Harnesses](https://mlsanity.com/paper/2603.28052/) (2026)
- [KernelBench: Can LLMs Write Efficient GPU Kernels?](https://mlsanity.com/paper/2502.10517/) (2025)
- [ShinkaEvolve: Towards Open-Ended And Sample-Efficient Program Evolution](https://mlsanity.com/paper/2509.19349/) (2025)
- [Learning to Discover at Test Time](https://mlsanity.com/paper/2601.16175/) (2026)
- [LLM-SRBench: A New Benchmark for Scientific Equation Discovery with Large Language Models](https://mlsanity.com/paper/2504.10415/) (2025)
- [ThetaEvolve: Test-time Learning on Open Problems](https://mlsanity.com/paper/2511.23473/) (2025)
- [AdaEvolve: Adaptive LLM Driven Zeroth-Order Optimization](https://mlsanity.com/paper/2602.20133/) (2026)
- [EvoX: Meta-Evolution for Automated Discovery](https://mlsanity.com/paper/2602.23413/) (2026)
- [ScienceBoard: Evaluating Multimodal Autonomous Agents in Realistic Scientific Workflows](https://mlsanity.com/paper/2505.19897/) (2025)
- [CORAL: Towards Autonomous Multi-Agent Evolution for Open-Ended Discovery](https://mlsanity.com/paper/2604.01658/) (2026)
- [CodeEvolve: an open source evolutionary coding agent for algorithmic discovery and optimization](https://mlsanity.com/paper/2510.14150/) (2025)
- [ALE-Bench: A Benchmark for Long-Horizon Objective-Driven Algorithm Engineering](https://mlsanity.com/paper/2506.09050/) (2025)
- [MoAI: Mixture of All Intelligence for Large Language and Vision Models](https://mlsanity.com/paper/2403.07508/) (2024)
- [Semi-Autonomous Mathematics Discovery with Gemini: A Case Study on the ErdHos Problems](https://mlsanity.com/paper/2601.22401/) (2026)
- [Meteor: Mamba-based Traversal of Rationale for Large Language and Vision Models](https://mlsanity.com/paper/2405.15574/) (2024)
- [AlgoTune: Can Language Models Speed Up General-Purpose Numerical Programs?](https://mlsanity.com/paper/2507.15887/) (2025)
- [CoLLaVO: Crayon Large Language and Vision mOdel](https://mlsanity.com/paper/2402.11248/) (2024)
- [GSO: Challenging Software Optimization Tasks for Evaluating SWE-Agents](https://mlsanity.com/paper/2505.23671/) (2025)
- [PACEvolve: Enabling Progress-Aware Consistent Evolution](https://mlsanity.com/paper/2601.10657/) (2026)
- [FrontierCS: Evolving Challenges for Evolving Intelligence](https://mlsanity.com/paper/2512.15699/) (2025)
- [Structured Scaling of AI Discovery Across Diverse Scientific Domains](https://mlsanity.com/paper/2604.19341/) (2026)
- [DialogCC: An Automated Pipeline for Creating High-Quality Multi-Modal Dialogue Dataset](https://mlsanity.com/paper/2212.04119/) (2022)
- [Phantom of Latent for Large Language and Vision Models](https://mlsanity.com/paper/2409.14713/) (2024)
- [SpatialClaw: Rethinking Action Interface for Agentic Spatial Reasoning](https://mlsanity.com/paper/2606.13673/) (2026)

## Built on by (most cited first, 6 of 6)

- [Mid-Harness: Scaling Actions Between Model and Harness for Terminal Agents](https://mlsanity.com/paper/2609.39982/) (2026)
- [EasyPPO: Stabilizing the Critic Is Key](https://mlsanity.com/paper/2609.36802/) (2026)
- [From Search to Signal: Online Post-Training in Automatic Heuristic Design](https://mlsanity.com/paper/2609.39383/) (2026)
- [EvoDuet: Bilevel Co-Evolution of Web Searching and Task Solving for Scientific Discovery](https://mlsanity.com/paper/2609.40340/) (2026)
- [Science or Slop?: Benchmarking and Mitigating Scientific Slop in AI-Generated Papers](https://mlsanity.com/paper/2610.00531/) (2026)
- [HarnessTTT: Learn to Optimize Harness at Test-Time for Scientific Discovery](https://mlsanity.com/paper/2610.17370/) (2026)

## Abstract (arXiv)

Would experience designing faster GPU kernels also help close in on a long-standing open mathematical conjecture? Large Language Models (LLMs) integrated into evolutionary search have recently produced state-of-the-art solutions on optimization tasks, including open mathematical conjectures, GPU kernel design, scientific law discovery, and combinatorial puzzles. To achieve this, prior work applied search scaffolds to one target task at a time, so every new problem is approached from scratch and the experience accumulated during search is discarded once the model finishes its attempt. This leaves the capability of iteratively evolving a solution (e.g., knowing which part to mutate and how, deciding when to backtrack) entirely in the scaffold rather than in the model itself. Whether the model itself could acquire this capability and reuse it across different tasks has been largely unexamined. To address this, we introduce Evolution Fine-Tuning (EFT), a mid-training paradigm that teaches LLMs to evolve solutions across tasks by converting evolutionary search trajectories into supervision. We construct Finch Collection, a 156K-trajectory dataset spanning 10 domains and 371 optimization tasks, and fine-tune open-source LLMs from 2B to 9B parameters. Empirically, EFT confers cross-task generalization: across 22 held-out tasks, our models surpass their base counterparts by 10.22% on average. Furthermore, when paired with test-time RL, our model matches state-of-the-art performance on two circle-packing tasks and outperforms its base-model counterpart on the ErdHos minimum-overlap problem. EFT thus serves as a "practice phase" for general-purpose discovery agents that do not solve new problems from scratch.