# AVO: Agentic Variation Operators for Autonomous Evolutionary Search

arXiv: https://arxiv.org/abs/2603.24517 · HTML: https://mlsanity.com/paper/2603.24517/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- Placed in General LLM agent frameworks and benchmarks, one of 16 directions in LLM agents and tool use.
- Builds on 17 of its 23 references that are in the map.
- First posted on arXiv 25 Mar 2026.

## Placed in

- LLM agents and tool use / [General LLM agent frameworks and benchmarks](https://mlsanity.com/agents/general-llm-agent-frameworks-and-benchmarks/)

## Builds on (most cited first, 17 of 17)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness](https://mlsanity.com/paper/2205.14135/) (2022)
- [SWE-bench: Can Language Models Resolve Real-World GitHub Issues?](https://mlsanity.com/paper/2310.06770/) (2023)
- [FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning](https://mlsanity.com/paper/2307.08691/) (2023)
- [SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering](https://mlsanity.com/paper/2405.15793/) (2024)
- [OpenHands: An Open Platform for AI Software Developers as Generalist Agents](https://mlsanity.com/paper/2407.16741/) (2024)
- [AlphaEvolve: A coding agent for scientific and algorithmic discovery](https://mlsanity.com/paper/2506.13131/) (2025)
- [FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precision](https://mlsanity.com/paper/2407.08608/) (2024)
- [Illuminating search spaces by mapping elites](https://mlsanity.com/paper/1504.04909/) (2015)
- [ReEvo: Large Language Models as Hyper-Heuristics with Reflective Evolution](https://mlsanity.com/paper/2402.01145/) (2024)
- [cuDNN: Efficient Primitives for Deep Learning](https://mlsanity.com/paper/1410.0759/) (2014)
- [Evolution through Large Models](https://mlsanity.com/paper/2206.08896/) (2022)
- [EvoPrompting: Language Models for Code-Level Neural Architecture Search](https://mlsanity.com/paper/2302.14838/) (2023)
- [Learning to Discover at Test Time](https://mlsanity.com/paper/2601.16175/) (2026)
- [FlashAttention-4: Algorithm and Kernel Pipelining Co-Design for Asymmetric Hardware Scaling](https://mlsanity.com/paper/2603.05451/) (2026)
- [LoongFlow: Directed Evolutionary Search via a Cognitive Plan-Execute-Summarize Paradigm](https://mlsanity.com/paper/2512.24077/) (2025)
- [VibeTensor: System Software for Deep Learning, Fully Generated by AI Agents](https://mlsanity.com/paper/2601.16238/) (2026)

## Built on by (most cited first, 24 of 24)

- [MLS-Bench: A Holistic and Rigorous Assessment of AI Systems on Building Better AI](https://mlsanity.com/paper/2605.08678/) (2026)
- [SwarmResearch: Orchestrating Coding Agents for Open-Ended Discovery](https://mlsanity.com/paper/2607.02807/) (2026)
- [CAKE: Compiler-Agent Co-Design for Frontier Kernel Evolution](https://mlsanity.com/paper/2608.12629/) (2026)
- [Optimizing CUDA like a Human: Micro-Profiling Tools as Expert Surrogates for LLM-Based GPU Kernel Optimization](https://mlsanity.com/paper/2606.26453/) (2026)
- [Pioneer Agent: Continual Improvement of Small Language Models in Production](https://mlsanity.com/paper/2604.09791/) (2026)
- [Xe-Forge: Multi-Stage LLM-Powered Kernel Optimization for Intel GPU](https://mlsanity.com/paper/2605.26118/) (2026)
- [Harness Engineering for LLM-Driven GPU Kernel Generation](https://mlsanity.com/paper/2607.17979/) (2026)
- [Hawk: Harnessing Hardware-Aware Knowledge for High-Performance NPU Kernel Generation](https://mlsanity.com/paper/2607.01590/) (2026)
- [SLAI T-Rex: Full-Parameter Post-training of the DeepSeek-V4 Family on Ascend SuperPOD](https://mlsanity.com/paper/2607.20145/) (2026)
- [WIDE: Boosting Adaptive LLM Inference via Token-level Dynamic Width Pruning](https://mlsanity.com/paper/2607.28418/) (2026)
- [PhyAI: Real-Time Physical AI at the Edge, Scalable Rollouts in the Cloud](https://mlsanity.com/paper/2608.03682/) (2026)
- [Ave: Guiding Agentic GPU Optimization Using Data-Flow Invariants](https://mlsanity.com/paper/2604.18616/) (2026)
- [CUDABeaver: Benchmarking LLM-Based Automated CUDA Debugging](https://mlsanity.com/paper/2605.08455/) (2026)
- [Autonomous Discovery of Wireless Communications Algorithms](https://mlsanity.com/paper/2607.17762/) (2026)
- [Compiler-Grounded Hierarchical Diagnosis for LLM-Based Triton Kernel Optimization](https://mlsanity.com/paper/2607.23089/) (2026)
- [KernelGenBench: Can LLMs and Agents Write Efficient Kernels Across Operator Sources and Hardware Platforms?](https://mlsanity.com/paper/2607.27231/) (2026)
- [KernelBrain: Coarse-to-Fine, Budget-Aware Search for Agentic GPU Kernel Optimization](https://mlsanity.com/paper/2608.02611/) (2026)
- [AsmEvo: Agentic Assembly-Level Optimization of AMD GPU Kernels with Functional Equivalence Verification](https://mlsanity.com/paper/2608.20711/) (2026)
- [AMDKernelVault: Large-Scale Datasets and Agentic Training for AMD GPU Kernel Optimization](https://mlsanity.com/paper/2609.12471/) (2026)
- [From PyTorch to the NPU: LLM-Agent-Driven Model Conversion Across Heterogeneous Inference Runtimes](https://mlsanity.com/paper/2609.27249/) (2026)
- [GPUPhysBench: Benchmarking Coding Agents for Correct and Efficient GPU Physics Simulation](https://mlsanity.com/paper/2609.35639/) (2026)
- [MetaKernelBench: Measuring GPU Kernel Knowledge Transfer Beyond Code](https://mlsanity.com/paper/2610.05014/) (2026)
- [VeriFine: Scaling Verification for Self-Improvement in Embodied Reasoning](https://mlsanity.com/paper/2610.08761/) (2026)
- [DuoDiscover: Scaling Autonomous Discovery Beyond the Evaluation Bottleneck](https://mlsanity.com/paper/2610.17184/) (2026)

## Abstract (arXiv)

Agentic Variation Operators (AVO) are a new family of evolutionary variation operators that replace the fixed mutation, crossover, and hand-designed heuristics of classical evolutionary search with autonomous coding agents. Rather than confining a language model to candidate generation within a prescribed pipeline, AVO instantiates variation as a self-directed agent loop that can consult the current lineage, a domain-specific knowledge base, and execution feedback to propose, repair, critique, and verify implementation edits. We evaluate AVO on attention, among the most aggressively optimized kernel targets in AI, on NVIDIA Blackwell (B200) GPUs. Over 7 days of continuous autonomous evolution on multi-head attention, AVO discovers kernels that outperform cuDNN by up to 3.5% and FlashAttention-4 by up to 10.5% across the evaluated configurations. The discovered optimizations transfer readily to grouped-query attention, requiring only 30 minutes of additional autonomous adaptation and yielding gains of up to 7.0% over cuDNN and 9.3% over FlashAttention-4. Together, these results show that agentic variation operators move beyond prior LLM-in-the-loop evolutionary pipelines by elevating the agent from candidate generator to variation operator, and can discover performance-critical micro-architectural optimizations that produce kernels surpassing state-of-the-art expert-engineered attention implementations on today's most advanced GPU hardware.