# NN-Former: Rethinking Graph Structure in Neural Architecture Representation

arXiv: https://arxiv.org/abs/2507.00880 · HTML: https://mlsanity.com/paper/2507.00880/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 43 of its 61 references that are in the map.
- First posted on arXiv 1 Jul 2025.

## Builds on (most cited first, 30 of 43)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Semi-Supervised Classification with Graph Convolutional Networks](https://mlsanity.com/paper/1609.02907/) (2016)
- [Graph Attention Networks](https://mlsanity.com/paper/1710.10903/) (2017)
- [Inductive Representation Learning on Large Graphs](https://mlsanity.com/paper/1706.02216/) (2017)
- [SGDR: Stochastic Gradient Descent with Warm Restarts](https://mlsanity.com/paper/1608.03983/) (2016)
- [How Powerful are Graph Neural Networks?](https://mlsanity.com/paper/1810.00826/) (2018)
- [Going Deeper with Convolutions](https://mlsanity.com/paper/1409.4842/) (2014)
- [Neural Message Passing for Quantum Chemistry](https://mlsanity.com/paper/1704.01212/) (2017)
- [Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour](https://mlsanity.com/paper/1706.02677/) (2017)
- [Once-for-All: Train One Network and Specialize it for Efficient Deployment](https://mlsanity.com/paper/1908.09791/) (2019)
- [A Generalization of Transformer Networks to Graphs](https://mlsanity.com/paper/2012.09699/) (2020)
- [RepVGG: Making VGG-style ConvNets Great Again](https://mlsanity.com/paper/2101.03697/) (2021)
- [Progressive Neural Architecture Search](https://mlsanity.com/paper/1712.00559/) (2017)
- [NAS-Bench-201: Extending the Scope of Reproducible Neural Architecture Search](https://mlsanity.com/paper/2001.00326/) (2020)
- [NAS-Bench-101: Towards Reproducible Neural Architecture Search](https://mlsanity.com/paper/1902.09635/) (2019)
- [CMT: Convolutional Neural Networks Meet Vision Transformers](https://mlsanity.com/paper/2107.06263/) (2021)
- [Pay Less Attention with Lightweight and Dynamic Convolutions](https://mlsanity.com/paper/1901.10430/) (2019)
- [Zero-Cost Proxies for Lightweight NAS](https://mlsanity.com/paper/2101.08134/) (2021)
- [Representing Long-Range Context for Graph Neural Networks with Global Attention](https://mlsanity.com/paper/2201.08821/) (2022)
- [BANANAS: Bayesian Optimization with Neural Architectures for Neural Architecture Search](https://mlsanity.com/paper/1910.11858/) (2019)
- [Neural Architecture Optimization](https://mlsanity.com/paper/1808.07233/) (2018)
- [Graph HyperNetworks for Neural Architecture Search](https://mlsanity.com/paper/1810.05749/) (2018)
- [Neural Predictor for Neural Architecture Search](https://mlsanity.com/paper/1912.00848/) (2019)
- [BRP-NAS: Prediction-based NAS using GCNs](https://mlsanity.com/paper/2007.08668/) (2020)
- [Directed Acyclic Graph Neural Networks](https://mlsanity.com/paper/2101.07965/) (2021)
- [Graph Structure of Neural Networks](https://mlsanity.com/paper/2007.06559/) (2020)
- [Does Unsupervised Architecture Representation Learning Help Neural Architecture Search?](https://mlsanity.com/paper/2006.06936/) (2020)
- [A Generic Graph-based Neural Architecture Encoding Scheme for Predictor-based NAS](https://mlsanity.com/paper/2004.01899/) (2020)
- [Semi-Supervised Neural Architecture Search](https://mlsanity.com/paper/2002.10389/) (2020)

## Built on by (most cited first, 3 of 3)

- [Node-wise Feature Encoding for Neural Performance Prediction](https://mlsanity.com/paper/2608.27794/) (2026)
- [RiPPLE: Cross-Space Performance Prediction from Early Training for Neural Architecture Search](https://mlsanity.com/paper/2609.12418/) (2026)
- [DTA-GT: Direction- and Topology-Aware Graph Transformer for Neural Network Representation Learning](https://mlsanity.com/paper/2610.16822/) (2026)

## Abstract (arXiv)

The growing use of deep learning necessitates efficient network design and deployment, making neural predictors vital for estimating attributes such as accuracy and latency. Recently, Graph Neural Networks (GNNs) and transformers have shown promising performance in representing neural architectures. However, each of both methods has its disadvantages. GNNs lack the capabilities to represent complicated features, while transformers face poor generalization when the depth of architecture grows. To mitigate the above issues, we rethink neural architecture topology and show that sibling nodes are pivotal while overlooked in previous research. We thus propose a novel predictor leveraging the strengths of GNNs and transformers to learn the enhanced topology. We introduce a novel token mixer that considers siblings, and a new channel mixer named bidirectional graph isomorphism feed-forward network. Our approach consistently achieves promising performance in both accuracy and latency prediction, providing valuable insights for learning Directed Acyclic Graph (DAG) topology. The code is available at https://github.com/XuRuihan/NNFormer.