# Modern Evolution Strategies for Creativity: Fitting Concrete Images and Abstract Concepts

arXiv: https://arxiv.org/abs/2109.08857 · HTML: https://mlsanity.com/paper/2109.08857/

## Key facts

- Cited by 40 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 57 references that are in the map.
- First posted on arXiv 18 Sep 2021.

## Builds on (most cited first, 22 of 22)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision](https://mlsanity.com/paper/2102.05918/) (2021)
- [Evolution Strategies as a Scalable Alternative to Reinforcement Learning](https://mlsanity.com/paper/1703.03864/) (2017)
- [Modular Primitives for High-Performance Differentiable Rendering](https://mlsanity.com/paper/2011.03277/) (2020)
- [A Neural Representation of Sketch Drawings](https://mlsanity.com/paper/1704.03477/) (2017)
- [DRAW: A Recurrent Neural Network For Image Generation](https://mlsanity.com/paper/1502.04623/) (2015)
- [CLIPDraw: Exploring Text-to-Drawing Synthesis through Language-Image Encoders](https://mlsanity.com/paper/2106.14843/) (2021)
- [Differentiable Rendering: A Survey](https://mlsanity.com/paper/2006.12057/) (2020)
- [A Learned Representation for Scalable Vector Graphics](https://mlsanity.com/paper/1904.02632/) (2019)
- [Synthesizing Programs for Images using Reinforced Adversarial Learning](https://mlsanity.com/paper/1804.01118/) (2018)
- [Learning to Paint With Model-based Deep Reinforcement Learning](https://mlsanity.com/paper/1903.04411/) (2019)
- [Paint Transformer: Feed Forward Neural Painting with Stroke Prediction](https://mlsanity.com/paper/2108.03798/) (2021)
- [Generating images from caption and vice versa via CLIP-Guided Generative Latent Space Search](https://mlsanity.com/paper/2102.01645/) (2021)
- [Neural Painters: A learned differentiable constraint for generating brushstroke paintings](https://mlsanity.com/paper/1904.08410/) (2019)
- [Unsupervised Doodling and Painting with Improved SPIRAL](https://mlsanity.com/paper/1910.01007/) (2019)
- [Sketchforme: Composing Sketched Scenes from Text Descriptions for Interactive Applications](https://mlsanity.com/paper/1904.04399/) (2019)
- [Scones: Towards Conversational Authoring of Sketches](https://mlsanity.com/paper/2005.07781/) (2020)
- [ClipUp: A Simple and Powerful Optimizer for Distribution-based Policy Evolution](https://mlsanity.com/paper/2008.02387/) (2020)
- [Generative Art Using Neural Visual Grammars and Dual Encoders](https://mlsanity.com/paper/2105.00162/) (2021)
- [Shared Visual Abstractions](https://mlsanity.com/paper/1912.04217/) (2019)
- [Evolved Art with Transparent, Overlapping, and Geometric Shapes](https://mlsanity.com/paper/1904.06110/) (2019)

## Built on by (most cited first, 30 of 40)

- [VQGAN-CLIP: Open Domain Image Generation and Editing with Natural Language Guidance](https://mlsanity.com/paper/2204.08583/) (2022)
- [VectorFusion: Text-to-SVG by Abstracting Pixel-Based Diffusion Models](https://mlsanity.com/paper/2211.11319/) (2022)
- [SVGDreamer: Text Guided SVG Generation with Diffusion Model](https://mlsanity.com/paper/2312.16476/) (2023)
- [OmniSVG: A Unified Scalable Vector Graphics Generation Model](https://mlsanity.com/paper/2504.06263/) (2025)
- [Empowering LLMs to Understand and Generate Complex Vector Graphics](https://mlsanity.com/paper/2412.11102/) (2024)
- [Word-As-Image for Semantic Typography](https://mlsanity.com/paper/2303.01818/) (2023)
- [Concept Decomposition for Visual Exploration and Inspiration](https://mlsanity.com/paper/2305.18203/) (2023)
- [CLIPasso: Semantically-Aware Object Sketching](https://mlsanity.com/paper/2202.05822/) (2022)
- [DiffSketcher: Text Guided Vector Sketch Synthesis through Latent Diffusion Models](https://mlsanity.com/paper/2306.14685/) (2023)
- [Renderable Neural Radiance Map for Visual Navigation](https://mlsanity.com/paper/2303.00304/) (2023)
- [CLIPVG: Text-Guided Image Manipulation Using Differentiable Vector Graphics](https://mlsanity.com/paper/2212.02122/) (2022)
- [StyleCLIPDraw: Coupling Content and Style in Text-to-Drawing Translation](https://mlsanity.com/paper/2202.12362/) (2022)
- [Automating the Search for Artificial Life with Foundation Models](https://mlsanity.com/paper/2412.17799/) (2024)
- [NeuralSVG: An Implicit Representation for Text-to-Vector Generation](https://mlsanity.com/paper/2501.03992/) (2025)
- [SVGenius: Benchmarking LLMs in SVG Understanding, Editing and Generation](https://mlsanity.com/paper/2506.03139/) (2025)
- [CLIP-CLOP: CLIP-Guided Collage and Photomontage](https://mlsanity.com/paper/2205.03146/) (2022)
- [Prompt Evolution for Generative AI: A Classifier-Guided Approach](https://mlsanity.com/paper/2305.16347/) (2023)
- [Text-Guided Vector Graphics Customization](https://mlsanity.com/paper/2309.12302/) (2023)
- [Stroke-based Rendering: From Heuristics to Deep Learning](https://mlsanity.com/paper/2302.00595/) (2022)
- [Primal and Dual Analysis of Entropic Fictitious Play for Finite-sum Problems](https://mlsanity.com/paper/2303.02957/) (2023)
- [NeuroEvoBench: Benchmarking Evolutionary Optimizers for Deep Learning Applications](https://mlsanity.com/paper/2311.02394/) (2023)
- [Creative Text-to-Audio Generation via Synthesizer Programming](https://mlsanity.com/paper/2406.00294/) (2024)
- [WIR3D: Visually-Informed and Geometry-Aware 3D Shape Abstraction](https://mlsanity.com/paper/2505.04813/) (2025)
- [Text-guided Image-and-Shape Editing and Generation: A Short Survey](https://mlsanity.com/paper/2304.09244/) (2023)
- [SVGFusion: A VAE-Diffusion Transformer for Vector Graphic Generation](https://mlsanity.com/paper/2412.10437/) (2024)
- [OmniLottie: Generating Vector Animations via Parameterized Lottie Tokens](https://mlsanity.com/paper/2603.02138/) (2026)
- [Bridging the prosody GAP: Genetic Algorithm with People to efficiently sample emotional prosody](https://mlsanity.com/paper/2205.04820/) (2022)
- [Inference-Time Alignment of Diffusion Models via Evolutionary Algorithms](https://mlsanity.com/paper/2506.00299/) (2025)
- [Evolving Three Dimension (3D) Abstract Art: Fitting Concepts by Language](https://mlsanity.com/paper/2304.12932/) (2023)
- [Solving Inverse Problems in Stochastic Self-Organizing Systems through Invariant Representations](https://mlsanity.com/paper/2506.11796/) (2025)

## Abstract (arXiv)

Evolutionary algorithms have been used in the digital art scene since the 1970s. A popular application of genetic algorithms is to optimize the procedural placement of vector graphic primitives to resemble a given painting. In recent years, deep learning-based approaches have also been proposed to generate procedural drawings, which can be optimized using gradient descent. In this work, we revisit the use of evolutionary algorithms for computational creativity. We find that modern evolution strategies (ES) algorithms, when tasked with the placement of shapes, offer large improvements in both quality and efficiency compared to traditional genetic algorithms, and even comparable to gradient-based methods. We demonstrate that ES is also well suited at optimizing the placement of shapes to fit the CLIP model, and can produce diverse, distinct geometric abstractions that are aligned with human interpretation of language. Videos and demo: https://es-clip.github.io/