# TART: Token-based Architecture Transformer for Neural Network Performance Prediction

arXiv: https://arxiv.org/abs/2501.02007 · HTML: https://mlsanity.com/paper/2501.02007/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 21 of its 29 references that are in the map.
- First posted on arXiv 2 Jan 2025.

## Builds on (most cited first, 21 of 21)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Generative Adversarial Networks](https://mlsanity.com/paper/2203.00667/) (2022)
- [DARTS: Differentiable Architecture Search](https://mlsanity.com/paper/1806.09055/) (2018)
- [Graph Neural Networks: A Review of Methods and Applications](https://mlsanity.com/paper/1812.08434/) (2018)
- [Efficient Neural Architecture Search via Parameter Sharing](https://mlsanity.com/paper/1802.03268/) (2018)
- [Invariant and Equivariant Graph Networks](https://mlsanity.com/paper/1812.09902/) (2018)
- [A Comprehensive Survey of Neural Architecture Search: Challenges and Solutions](https://mlsanity.com/paper/2006.02903/) (2020)
- [SMASH: One-Shot Model Architecture Search through HyperNetworks](https://mlsanity.com/paper/1708.05344/) (2017)
- [Pure Transformers are Powerful Graph Learners](https://mlsanity.com/paper/2207.02505/) (2022)
- [TransGAN: Two Pure Transformers Can Make One Strong GAN, and That Can Scale Up](https://mlsanity.com/paper/2102.07074/) (2021)
- [StyleSwin: Transformer-based GAN for High-resolution Image Generation](https://mlsanity.com/paper/2112.10762/) (2021)
- [Graph HyperNetworks for Neural Architecture Search](https://mlsanity.com/paper/1810.05749/) (2018)
- [A Survey of Controllable Text Generation using Transformer-based Pre-trained Language Models](https://mlsanity.com/paper/2201.05337/) (2022)
- [Evaluating the Search Phase of Neural Architecture Search](https://mlsanity.com/paper/1902.08142/) (2019)
- [Generative Adversarial Transformers](https://mlsanity.com/paper/2103.01209/) (2021)
- [Neural Predictor for Neural Architecture Search](https://mlsanity.com/paper/1912.00848/) (2019)
- [AutoGAN: Neural Architecture Search for Generative Adversarial Networks](https://mlsanity.com/paper/1908.03835/) (2019)
- [How Powerful are Performance Predictors in Neural Architecture Search?](https://mlsanity.com/paper/2104.01177/) (2021)
- [Parameter Prediction for Unseen Deep Architectures](https://mlsanity.com/paper/2110.13100/) (2021)
- [Directed Acyclic Graph Neural Networks](https://mlsanity.com/paper/2101.07965/) (2021)
- [Improving Visual Quality of Image Synthesis by A Token-based Generator with Transformers](https://mlsanity.com/paper/2111.03481/) (2021)

## Abstract (arXiv)

In the realm of neural architecture design, achieving high performance is largely reliant on the manual expertise of researchers. Despite the emergence of Neural Architecture Search (NAS) as a promising technique for automating this process, current NAS methods still require human input to expand the search space and cannot generate new architectures. This paper explores the potential of Transformers in comprehending neural architectures and their performance, with the objective of establishing the foundation for utilizing Transformers to generate novel networks. We propose the Token-based Architecture Transformer (TART), which predicts neural network performance without the need to train candidate networks. TART attains state-of-the-art performance on the DeepNets-1M dataset for performance prediction tasks without edge information, indicating the potential of Transformers to aid in discovering novel and high-performing neural architectures.