# Language Embedding Meets Dynamic Graph: A New Exploration for Neural Architecture Representation Learning

arXiv: https://arxiv.org/abs/2506.07735 · HTML: https://mlsanity.com/paper/2506.07735/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 28 of its 42 references that are in the map.
- First posted on arXiv 9 Jun 2025.

## Builds on (most cited first, 28 of 28)

- [Semi-Supervised Classification with Graph Convolutional Networks](https://mlsanity.com/paper/1609.02907/) (2016)
- [Graph Attention Networks](https://mlsanity.com/paper/1710.10903/) (2017)
- [How Powerful are Graph Neural Networks?](https://mlsanity.com/paper/1810.00826/) (2018)
- [DARTS: Differentiable Architecture Search](https://mlsanity.com/paper/1806.09055/) (2018)
- [Neural Architecture Search with Reinforcement Learning](https://mlsanity.com/paper/1611.01578/) (2016)
- [Once-for-All: Train One Network and Specialize it for Efficient Deployment](https://mlsanity.com/paper/1908.09791/) (2019)
- [Benchmarking Graph Neural Networks](https://mlsanity.com/paper/2003.00982/) (2020)
- [ProxylessNAS: Direct Neural Architecture Search on Target Task and Hardware](https://mlsanity.com/paper/1812.00332/) (2018)
- [Representation Learning on Graphs: Methods and Applications](https://mlsanity.com/paper/1709.05584/) (2017)
- [NAS-Bench-201: Extending the Scope of Reproducible Neural Architecture Search](https://mlsanity.com/paper/2001.00326/) (2020)
- [NAS-Bench-101: Towards Reproducible Neural Architecture Search](https://mlsanity.com/paper/1902.09635/) (2019)
- [Representing Long-Range Context for Graph Neural Networks with Global Attention](https://mlsanity.com/paper/2201.08821/) (2022)
- [BANANAS: Bayesian Optimization with Neural Architectures for Neural Architecture Search](https://mlsanity.com/paper/1910.11858/) (2019)
- [Understanding and Robustifying Differentiable Architecture Search](https://mlsanity.com/paper/1909.09656/) (2019)
- [Neural Architecture Optimization](https://mlsanity.com/paper/1808.07233/) (2018)
- [Neural Predictor for Neural Architecture Search](https://mlsanity.com/paper/1912.00848/) (2019)
- [BRP-NAS: Prediction-based NAS using GCNs](https://mlsanity.com/paper/2007.08668/) (2020)
- [Does Unsupervised Architecture Representation Learning Help Neural Architecture Search?](https://mlsanity.com/paper/2006.06936/) (2020)
- [A Generic Graph-based Neural Architecture Encoding Scheme for Predictor-based NAS](https://mlsanity.com/paper/2004.01899/) (2020)
- [Semi-Supervised Neural Architecture Search](https://mlsanity.com/paper/2002.10389/) (2020)
- [ReNAS:Relativistic Evaluation of Neural Architecture Search](https://mlsanity.com/paper/1910.01523/) (2019)
- [Contrastive Neural Architecture Search with Neural Architecture Comparators](https://mlsanity.com/paper/2103.05471/) (2021)
- [NAS-Bench-Graph: Benchmarking Graph Neural Architecture Search](https://mlsanity.com/paper/2206.09166/) (2022)
- [Peephole: Predicting Network Performance Before Training](https://mlsanity.com/paper/1712.03351/) (2017)
- [Transformers over Directed Acyclic Graphs](https://mlsanity.com/paper/2210.13148/) (2022)
- [PACE: A Parallelizable Computation Encoder for Directed Acyclic Graphs](https://mlsanity.com/paper/2203.10304/) (2022)
- [NAR-Former: Neural Architecture Representation Learning towards Holistic Attributes Prediction](https://mlsanity.com/paper/2211.08024/) (2022)
- [NAR-Former V2: Rethinking Transformer for Universal Neural Network Representation Learning](https://mlsanity.com/paper/2306.10792/) (2023)

## Abstract (arXiv)

Neural Architecture Representation Learning aims to transform network models into feature representations for predicting network attributes, playing a crucial role in deploying and designing networks for real-world applications. Recently, inspired by the success of transformers, transformer-based models integrated with Graph Neural Networks (GNNs) have achieved significant progress in representation learning. However, current methods still have some limitations. First, existing methods overlook hardware attribute information, which conflicts with the current trend of diversified deep learning hardware and limits the practical applicability of models. Second, current encoding approaches rely on static adjacency matrices to represent topological structures, failing to capture the structural differences between computational nodes, which ultimately compromises encoding effectiveness. In this paper, we introduce LeDG-Former, an innovative framework that addresses these limitations through the synergistic integration of language-based semantic embedding and dynamic graph representation learning. Specifically, inspired by large language models (LLMs), we propose a language embedding framework where both neural architectures and hardware platform specifications are projected into a unified semantic space through tokenization and LLM processing, enabling zero-shot prediction across different hardware platforms for the first time. Then, we propose a dynamic graph-based transformer for modeling neural architectures, resulting in improved neural architecture modeling performance. On the NNLQP benchmark, LeDG-Former surpasses previous methods, establishing a new SOTA while demonstrating the first successful cross-hardware latency prediction capability. Furthermore, our framework achieves superior performance on the cell-structured NAS-Bench-101 and NAS-Bench-201 datasets.