# Deep-ELA: Deep Exploratory Landscape Analysis with Self-Supervised Pretrained Transformers for Single- and Multi-Objective Continuous Optimization Problems

arXiv: https://arxiv.org/abs/2401.01192 · HTML: https://mlsanity.com/paper/2401.01192/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 65 references that are in the map.
- First posted on arXiv 2 Jan 2024.

## Builds on (most cited first, 22 of 22)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [Bootstrap your own latent: A new approach to self-supervised Learning](https://mlsanity.com/paper/2006.07733/) (2020)
- [An Empirical Study of Training Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.02057/) (2021)
- [Transformer Feed-Forward Layers Are Key-Value Memories](https://mlsanity.com/paper/2012.14913/) (2020)
- [PCT: Point cloud transformer](https://mlsanity.com/paper/2012.09688/) (2020)
- [Language Modeling with Gated Convolutional Networks](https://mlsanity.com/paper/1612.08083/) (2016)
- [SMAC3: A Versatile Bayesian Optimization Package for Hyperparameter Optimization](https://mlsanity.com/paper/2109.09831/) (2021)
- [Transformers without Tears: Improving the Normalization of Self-Attention](https://mlsanity.com/paper/1910.05895/) (2019)
- [Automated Algorithm Selection: Survey and Perspectives](https://mlsanity.com/paper/1811.11597/) (2018)
- [A Survey of Methods for Automated Algorithm Configuration](https://mlsanity.com/paper/2202.01651/) (2022)
- [Comprehensive Feature-Based Landscape Analysis of Continuous and Constrained Optimization Problems Using the R-Package flacco](https://mlsanity.com/paper/1708.05258/) (2017)
- [Exploratory Landscape Analysis is Strongly Sensitive to the Sampling Strategy](https://mlsanity.com/paper/2006.11135/) (2020)
- [DoE2Vec: Deep-learning Based Features for Exploratory Landscape Analysis](https://mlsanity.com/paper/2304.01219/) (2023)
- [Deep Learning as a Competitive Feature-Free Approach for Automated Algorithm Selection on the Traveling Salesperson Problem](https://mlsanity.com/paper/2006.15968/) (2020)
- [Using Well-Understood Single-Objective Functions in Multiobjective Black-Box Optimization Test Suites](https://mlsanity.com/paper/1604.00359/) (2016)
- [A Collection of Deep Learning-based Feature-Free Approaches for Characterizing Single-Objective Continuous Fitness Landscapes](https://mlsanity.com/paper/2204.05752/) (2022)
- [Linear Matrix Factorization Embeddings for Single-objective Optimization Landscapes](https://mlsanity.com/paper/2009.14506/) (2020)
- [TransOpt: Transformer-based Representation Learning for Optimization Problem Classification](https://mlsanity.com/paper/2311.18035/) (2023)
- [Tools for Landscape Analysis of Optimisation Problems in Procedural Content Generation for Games](https://mlsanity.com/paper/2302.08479/) (2023)

## Built on by (most cited first, 9 of 9)

- [Toward Automated Algorithm Design: A Survey and Practical Guide to Meta-Black-Box-Optimization](https://mlsanity.com/paper/2411.00625/) (2024)
- [Neural Exploratory Landscape Analysis for Meta-Black-Box-Optimization](https://mlsanity.com/paper/2408.10672/) (2024)
- [Learning Where, What and How to Transfer: A Multi-Role Reinforcement Learning Approach for Evolutionary Multitasking](https://mlsanity.com/paper/2511.15199/) (2025)
- [Beyond Numerical Features: CNN-Driven Algorithm Selection via Contour Plots for Continuous Black-Box Optimization](https://mlsanity.com/paper/2605.20797/) (2026)
- [From Performance to Understanding: A Vision for Explainable Automated Algorithm Design](https://mlsanity.com/paper/2511.16201/) (2025)
- [Geometric Probing for Algorithm Selection in Continuous Black-Box Optimization](https://mlsanity.com/paper/2604.09095/) (2026)
- [On the Influence of the Feature Computation Budget on Per-Instance Algorithm Selection for Black-Box Optimization](https://mlsanity.com/paper/2605.04954/) (2026)
- [Unsupervised Multi-kernel Learning for Automated Algorithm Selection](https://mlsanity.com/paper/2607.19031/) (2026)
- [Hyper Algorithm Design Agent: Evolving Learnable Optimizer from Zero](https://mlsanity.com/paper/2609.35328/) (2026)

## Abstract (arXiv)

In many recent works, the potential of Exploratory Landscape Analysis (ELA) features to numerically characterize, in particular, single-objective continuous optimization problems has been demonstrated. These numerical features provide the input for all kinds of machine learning tasks on continuous optimization problems, ranging, i.a., from High-level Property Prediction to Automated Algorithm Selection and Automated Algorithm Configuration. Without ELA features, analyzing and understanding the characteristics of single-objective continuous optimization problems is -- to the best of our knowledge -- very limited. Yet, despite their usefulness, as demonstrated in several past works, ELA features suffer from several drawbacks. These include, in particular, (1.) a strong correlation between multiple features, as well as (2.) its very limited applicability to multi-objective continuous optimization problems. As a remedy, recent works proposed deep learning-based approaches as alternatives to ELA. In these works, e.g., point-cloud transformers were used to characterize an optimization problem's fitness landscape. However, these approaches require a large amount of labeled training data. Within this work, we propose a hybrid approach, Deep-ELA, which combines (the benefits of) deep learning and ELA features. Specifically, we pre-trained four transformers on millions of randomly generated optimization problems to learn deep representations of the landscapes of continuous single- and multi-objective optimization problems. Our proposed framework can either be used out-of-the-box for analyzing single- and multi-objective continuous optimization problems, or subsequently fine-tuned to various tasks focussing on algorithm behavior and problem understanding.