# Architecture-Aware Learning Curve Extrapolation via Graph Ordinary Differential Equation

arXiv: https://arxiv.org/abs/2412.15554 · HTML: https://mlsanity.com/paper/2412.15554/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 45 references that are in the map.
- First posted on arXiv 20 Dec 2024.

## Builds on (most cited first, 30 of 30)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Semi-Supervised Classification with Graph Convolutional Networks](https://mlsanity.com/paper/1609.02907/) (2016)
- [Graph Attention Networks](https://mlsanity.com/paper/1710.10903/) (2017)
- [How Powerful are Graph Neural Networks?](https://mlsanity.com/paper/1810.00826/) (2018)
- [Neural Ordinary Differential Equations](https://mlsanity.com/paper/1806.07366/) (2018)
- [DARTS: Differentiable Architecture Search](https://mlsanity.com/paper/1806.09055/) (2018)
- [Hierarchical Graph Representation Learning with Differentiable Pooling](https://mlsanity.com/paper/1806.08804/) (2018)
- [Heterogeneous Graph Transformer](https://mlsanity.com/paper/2003.01332/) (2020)
- [OpenML: networked science in machine learning](https://mlsanity.com/paper/1407.7722/) (2014)
- [NAS-Bench-201: Extending the Scope of Reproducible Neural Architecture Search](https://mlsanity.com/paper/2001.00326/) (2020)
- [Neural Relational Inference for Interacting Systems](https://mlsanity.com/paper/1802.04687/) (2018)
- [Trained Transformers Learn Linear Models In-Context](https://mlsanity.com/paper/2306.09927/) (2023)
- [Gate-Variants of Gated Recurrent Unit (GRU) Neural Networks](https://mlsanity.com/paper/1701.05923/) (2017)
- [A Differential Equation for Modeling Nesterov's Accelerated Gradient Method: Theory and Insights](https://mlsanity.com/paper/1503.01243/) (2015)
- [Neural Predictor for Neural Architecture Search](https://mlsanity.com/paper/1912.00848/) (2019)
- [Freeze-Thaw Bayesian Optimization](https://mlsanity.com/paper/1406.3896/) (2014)
- [Parameter Prediction for Unseen Deep Architectures](https://mlsanity.com/paper/2110.13100/) (2021)
- [Learning Continuous System Dynamics from Irregularly-Sampled Partial Observations](https://mlsanity.com/paper/2011.03880/) (2020)
- [Does Unsupervised Architecture Representation Learning Help Neural Architecture Search?](https://mlsanity.com/paper/2006.06936/) (2020)
- [A Generic Graph-based Neural Architecture Encoding Scheme for Predictor-based NAS](https://mlsanity.com/paper/2004.01899/) (2020)
- [Efficient Bayesian Learning Curve Extrapolation using Prior-Data Fitted Networks](https://mlsanity.com/paper/2310.20447/) (2023)
- [A Semi-Supervised Assessor of Neural Architectures](https://mlsanity.com/paper/2005.06821/) (2020)
- [Speedy Performance Estimation for Neural Architecture Search](https://mlsanity.com/paper/2006.04492/) (2020)
- [CF-GODE: Continuous-Time Causal Inference for Multi-Agent Dynamical Systems](https://mlsanity.com/paper/2306.11216/) (2023)
- [Probabilistic Rollouts for Learning Curve Extrapolation Across Hyperparameter Settings](https://mlsanity.com/paper/1910.04522/) (2019)
- [Physics-Informed Regularization for Domain-Agnostic Dynamical System Modeling](https://mlsanity.com/paper/2410.06366/) (2024)
- [Bridging the Gap between Sample-based and One-shot Neural Architecture Search with BONAS](https://mlsanity.com/paper/1911.09336/) (2019)
- [Causal Graph ODE: Continuous Treatment Effect Modeling in Multi-agent Dynamical Systems](https://mlsanity.com/paper/2403.00178/) (2024)
- [A Variational-Sequential Graph Autoencoder for Neural Architecture Performance Prediction](https://mlsanity.com/paper/1912.05317/) (2019)
- [A Variational Perspective on High-Resolution ODEs](https://mlsanity.com/paper/2311.02002/) (2023)

## Built on by (most cited first, 5 of 5)

- [Graph ODEs and Beyond: A Comprehensive Survey on Integrating Differential Equations with Graph Neural Networks](https://mlsanity.com/paper/2503.23167/) (2025)
- [NNGPT: Rethinking AutoML with Large Language Models](https://mlsanity.com/paper/2511.20333/) (2025)
- [Gradient Flow Matching for Learning Update Dynamics in Neural Network Training](https://mlsanity.com/paper/2505.20221/) (2025)
- [Optimization as a Dynamical System: Generative Schedules from Latent ODEs](https://mlsanity.com/paper/2509.23052/) (2025)
- [AutoBCI: Forecast-Guided Agentic Neural Architecture Discovery for EEG-Based Brain--Computer Interfaces](https://mlsanity.com/paper/2609.35456/) (2026)

## Abstract (arXiv)

Learning curve extrapolation predicts neural network performance from early training epochs and has been applied to accelerate AutoML, facilitating hyperparameter tuning and neural architecture search. However, existing methods typically model the evolution of learning curves in isolation, neglecting the impact of neural network (NN) architectures, which influence the loss landscape and learning trajectories. In this work, we explore whether incorporating neural network architecture improves learning curve modeling and how to effectively integrate this architectural information. Motivated by the dynamical system view of optimization, we propose a novel architecture-aware neural differential equation model to forecast learning curves continuously. We empirically demonstrate its ability to capture the general trend of fluctuating learning curves while quantifying uncertainty through variational parameters. Our model outperforms current state-of-the-art learning curve extrapolation methods and pure time-series modeling approaches for both MLP and CNN-based learning curves. Additionally, we explore the applicability of our method in Neural Architecture Search scenarios, such as training configuration ranking.