# Multimodal LLM-assisted Evolutionary Search for Programmatic Control Policies

arXiv: https://arxiv.org/abs/2508.05433 · HTML: https://mlsanity.com/paper/2508.05433/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 38 of its 54 references that are in the map.
- First posted on arXiv 7 Aug 2025.

## Placed in

- Multimodal LLMs / vision-language models / [Embodied and 3D vision-language models](https://mlsanity.com/mllm/embodied-and-3d-vision-language-models/)

## Builds on (most cited first, 30 of 38)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Qwen2.5-VL Technical Report](https://mlsanity.com/paper/2502.13923/) (2025)
- [GPT-4o System Card](https://mlsanity.com/paper/2410.21276/) (2024)
- [Playing Atari with Deep Reinforcement Learning](https://mlsanity.com/paper/1312.5602/) (2013)
- [Eureka: Human-Level Reward Design via Coding Large Language Models](https://mlsanity.com/paper/2310.12931/) (2023)
- [ReEvo: Large Language Models as Hyper-Heuristics with Reflective Evolution](https://mlsanity.com/paper/2402.01145/) (2024)
- [Evolution of Heuristics: Towards Efficient Automatic Algorithm Design Using Large Language Model](https://mlsanity.com/paper/2401.02051/) (2024)
- [Reward Design with Language Models](https://mlsanity.com/paper/2303.00001/) (2023)
- [Programmatically Interpretable Reinforcement Learning](https://mlsanity.com/paper/1804.02477/) (2018)
- [Verifiable Reinforcement Learning via Policy Extraction](https://mlsanity.com/paper/1805.08328/) (2018)
- [Explainable Reinforcement Learning: A Survey](https://mlsanity.com/paper/2005.06247/) (2020)
- [The Revolution of Multimodal Large Language Models: A Survey](https://mlsanity.com/paper/2402.12451/) (2024)
- [LLM-SR: Scientific Equation Discovery via Programming with Large Language Models](https://mlsanity.com/paper/2404.18400/) (2024)
- [A Survey on Interpretable Reinforcement Learning](https://mlsanity.com/paper/2112.13112/) (2021)
- [Learning to Synthesize Programs as Interpretable and Generalizable Policies](https://mlsanity.com/paper/2108.13643/) (2021)
- [Neural Logic Reinforcement Learning](https://mlsanity.com/paper/1904.10729/) (2019)
- [Algorithm Evolution Using Large Language Model](https://mlsanity.com/paper/2311.15249/) (2023)
- [Interpretable Policies for Reinforcement Learning by Genetic Programming](https://mlsanity.com/paper/1712.04170/) (2017)
- [Interpretable and Explainable Logical Policies via Neurally Guided Symbolic Abstraction](https://mlsanity.com/paper/2306.01439/) (2023)
- [Evolving Code with A Large Language Model](https://mlsanity.com/paper/2401.07102/) (2024)
- [From Multimodal LLMs to Generalist Embodied Agents: Methods and Lessons](https://mlsanity.com/paper/2412.08442/) (2024)
- [PlanAgent: A Multi-modal Large Language Agent for Closed-loop Vehicle Motion Planning](https://mlsanity.com/paper/2406.01587/) (2024)
- [Explainable Deep Reinforcement Learning: State of the Art and Challenges](https://mlsanity.com/paper/2301.09937/) (2023)
- [A Large Language Model-Driven Reward Design Framework via Dynamic Feedback for Reinforcement Learning](https://mlsanity.com/paper/2410.14660/) (2024)
- [Neuro-Symbolic Hierarchical Rule Induction](https://mlsanity.com/paper/2112.13418/) (2021)
- [REvolve: Reward Evolution with Large Language Models using Human Feedback](https://mlsanity.com/paper/2406.01309/) (2024)
- [Visual Reasoning and Multi-Agent Approach in Multimodal Large Language Models (MLLMs): Solving TSP and mTSP Combinatorial Challenges](https://mlsanity.com/paper/2407.00092/) (2024)
- [Hierarchical Programmatic Reinforcement Learning via Learning to Compose Programs](https://mlsanity.com/paper/2301.12950/) (2023)
- [Interpretable and Editable Programmatic Tree Policies for Reinforcement Learning](https://mlsanity.com/paper/2405.14956/) (2024)

## Built on by (most cited first, 8 of 8)

- [MEMENTO: Memory-Guided Memetic Code-as-Policy Evolution](https://mlsanity.com/paper/2607.22832/) (2026)
- [REFLEX: Reflective Evolution from LLM Experience](https://mlsanity.com/paper/2606.16496/) (2026)
- [SpecAHD: Localize to Specialize for Automated Heuristic Design in Large-Scale Routing Problems](https://mlsanity.com/paper/2607.23676/) (2026)
- [Beyond Average Performance: Dynamic Instance Clustering and Specialized Algorithm Design in LLM-Assisted Evolutionary Search](https://mlsanity.com/paper/2608.03129/) (2026)
- [CDEoH: Category-Driven Automatic Algorithm Design With Large Language Models](https://mlsanity.com/paper/2603.19284/) (2026)
- [PACE: Primitive-Aware Code Evolution for Automated Algorithm Design](https://mlsanity.com/paper/2608.07395/) (2026)
- [Iterative Policy Refinement through Semantic Rollout Analysis](https://mlsanity.com/paper/2610.01652/) (2026)
- [TRACE: Tackling Real-World Resource Assignment Problems via Agentic Heuristic Design](https://mlsanity.com/paper/2610.01887/) (2026)

## Abstract (arXiv)

Deep reinforcement learning has achieved impressive success in control tasks. However, its policies, represented as opaque neural networks, are often difficult for humans to understand, verify, and debug, which undermines trust and hinders real-world deployment. This work addresses this challenge by introducing a novel approach for programmatic control policy discovery, called Multimodal Large Language Model-assisted Evolutionary Search (MLES). MLES utilizes multimodal large language models as programmatic policy generators, combining them with evolutionary search to automate policy generation. It integrates visual feedback-driven behavior analysis within the policy generation process to identify failure patterns and guide targeted improvements, thereby enhancing policy discovery efficiency and producing adaptable, human-aligned policies. Experimental results demonstrate that MLES achieves performance comparable to Proximal Policy Optimization (PPO) across two standard control tasks while providing transparent control logic and traceable design processes. This approach also overcomes the limitations of predefined domain-specific languages, facilitates knowledge transfer and reuse, and is scalable across various tasks, showing promise as a new paradigm for developing transparent and verifiable control policies. Code is publicly available at https://github.com/QingL2000/MLES.