mlsanityData to 7 Oct 2026

An algorithmic framework for the optimization of deep neural networks architectures and hyperparameters

arXiv 2303.12797 ↗First posted 27 Feb 2023cs.NE cs.AI cs.LGcited by 6

Not on the map. Why?

Abstract (arXiv)

In this paper, we propose an algorithmic framework to automatically generate efficient deep neural networks and optimize their associated hyperparameters. The framework is based on evolving directed acyclic graphs (DAGs), defining a more flexible search space than the existing ones in the literature. It allows mixtures of different classical operations: convolutions, recurrences and dense layers, but also more newfangled operations such as self-attention. Based on this search space we propose neighbourhood and evolution search operators to optimize both the architecture and hyper-parameters of our networks. These search operators can be used with any metaheuristic capable of handling mixed search spaces. We tested our algorithmic framework with an evolutionary algorithm on a time series prediction benchmark. The results demonstrate that our framework was able to find models outperforming the established baseline on numerous datasets.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Builds on 36 of its 56 references that are in the map. First posted on arXiv 27 Feb 2023.

Builds on

36 of 56 references.

References · 36

  1. 2017Attention Is All You Needcited by 59,932
  2. 2014Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translationcited by 4,016
  3. 2014Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modelingcited by 2,341
  4. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,830
  5. 2018DARTS: Differentiable Architecture Searchcited by 1,782
  6. 2016WaveNet: A Generative Model for Raw Audiocited by 1,710
  7. 2017Learning Transferable Architectures for Scalable Image Recognitioncited by 1,097
  8. 2017DeepAR: Probabilistic Forecasting with Autoregressive Recurrent Networkscited by 1,004
  9. 2018Efficient Neural Architecture Search via Parameter Sharingcited by 831
  10. 2019Temporal Fusion Transformers for Interpretable Multi-horizon Time Series Forecastingcited by 748
  11. 2019N-BEATS: Neural basis expansion analysis for interpretable time series forecastingcited by 744
  12. 2016Temporal Convolutional Networks for Action Segmentation and Detectioncited by 504
24 more
  1. 2019Single Path One-Shot Neural Architecture Search with Uniform Samplingcited by 411
  2. 2019NAS-Bench-101: Towards Reproducible Neural Architecture Searchcited by 340
  3. 2019On the Relationship between Self-Attention and Convolutional Layerscited by 299
  4. 2019Auto-DeepLab: Hierarchical Neural Architecture Search for Semantic Image Segmentationcited by 258
  5. 2021Monash Time Series Forecasting Archivecited by 256
  6. 2017SMASH: One-Shot Model Architecture Search through HyperNetworkscited by 200
  7. 2017Hierarchical Representations for Efficient Architecture Searchcited by 188
  8. 2019The Evolved Transformercited by 180
  9. 2019BANANAS: Bayesian Optimization with Neural Architectures for Neural Architecture Searchcited by 175
  10. 2023Neural Architecture Search: Insights from 1000 Paperscited by 136
  11. 2017Genetic CNNcited by 124
  12. 2019D-VAE: A Variational Autoencoder for Directed Acyclic Graphscited by 98
  13. 2019Understanding Architectures Learnt by Cell-based Neural Architecture Searchcited by 48
  14. 2021Nondeterminism and Instability in Neural Network Optimizationcited by 47
  15. 2018DENSER: Deep Evolutionary Network Structured Representationcited by 16
  16. 2021AutoAI-TS: AutoAI for Time Series Forecastingcited by 15
  17. 2022Efficient Automated Deep Learning for Time Series Forecastingcited by 14
  18. 2022Survey on Evolutionary Deep Learning: Principles, Algorithms, Applications and Open Issuescited by 12
  19. 2017A Particle Swarm Optimization-based Flexible Convolutional Auto-Encoder for Image Classificationcited by 11
  20. 2019Evolving Deep Neural Networks by Multi-objective Particle Swarm Optimization for Image Classificationcited by 11
  21. 2021Scale-Aware Neural Architecture Search for Multivariate Time Series Forecastingcited by 9
  22. 2020Bayesian Neural Architecture Search using A Training-Free Performance Metriccited by 8
  23. 2018Evolving Deep Convolutional Neural Networks by Variable-length Particle Swarm Optimization for Image Classificationcited by 7
  24. 2021Graph-based Neural Architecture Search with Operation Embeddingscited by 5

Built on by

Cited by year

Citing papers per year: 2024: 4; 2025: 2.

6 papers.

  1. 2024WindDragon: Enhancing wind power forecasting with Automated Deep Learningcited by 3
  2. 2024Automated Deep Learning for Load Forecastingcited by 3
  3. 2024A Bandit Approach with Evolutionary Operators for Model Selectioncited by 2
  4. 2024Automated Spatio-Temporal Weather Modeling for Load Forecastingcited by 1
  5. 2025AutoML Algorithms for Online Generalized Additive Model Selection: Application to Electricity Demand Forecastingcited by 0
  6. 2025Neural Architecture Search for global multi-step Forecasting of Energy Production Time Seriescited by 0