# TextNAS: A Neural Architecture Search Space tailored for Text Representation

arXiv: https://arxiv.org/abs/1912.10729 · HTML: https://mlsanity.com/paper/1912.10729/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 23 Dec 2019.

## Built on by (most cited first, 16 of 16)

- [AutoCTS: Automated Correlated Time Series Forecasting -- Extended Version](https://mlsanity.com/paper/2112.11174/) (2021)
- [A Survey of Techniques for Optimizing Transformer Inference](https://mlsanity.com/paper/2307.07982/) (2023)
- [Teacher-Student Architecture for Knowledge Distillation: A Survey](https://mlsanity.com/paper/2308.04268/) (2023)
- [Teacher-Student Architecture for Knowledge Learning: A Survey](https://mlsanity.com/paper/2210.17332/) (2022)
- [Rethinking Neural Operations for Diverse Tasks](https://mlsanity.com/paper/2103.15798/) (2021)
- [Neural Architecture Search on Efficient Transformers and Beyond](https://mlsanity.com/paper/2207.13955/) (2022)
- [SeqNAS: Neural Architecture Search for Event Sequence Classification](https://mlsanity.com/paper/2401.03246/) (2024)
- [Visionary: Vision architecture discovery for robot learning](https://mlsanity.com/paper/2103.14633/) (2021)
- [Modular Machine Learning: An Indispensable Path towards New-Generation Large Language Models](https://mlsanity.com/paper/2504.20020/) (2025)
- [Evolutionary Neural Architecture Search for Transformer in Knowledge Tracing](https://mlsanity.com/paper/2310.01180/) (2023)
- [Designing Novel Cognitive Diagnosis Models via Evolutionary Multi-Objective Neural Architecture Search](https://mlsanity.com/paper/2307.04429/) (2023)
- [pTNAS: Progressive Neural Architecture Search for Tabular Data](https://mlsanity.com/paper/2403.10318/) (2024)
- [DDNAS: Discretized Differentiable Neural Architecture Search for Text Classification](https://mlsanity.com/paper/2307.06005/) (2023)
- [Loss Functions for Predictor-based Neural Architecture Search](https://mlsanity.com/paper/2506.05869/) (2025)
- [HHNAS-AM: Hierarchical Hybrid Neural Architecture Search using Adaptive Mutation Policies](https://mlsanity.com/paper/2508.14946/) (2025)
- [Breaking Robustness Barriers in Cognitive Diagnosis: A One-Shot Neural Architecture Search Perspective](https://mlsanity.com/paper/2601.04918/) (2026)

## Abstract (arXiv)

Learning text representation is crucial for text classification and other language related tasks. There are a diverse set of text representation networks in the literature, and how to find the optimal one is a non-trivial problem. Recently, the emerging Neural Architecture Search (NAS) techniques have demonstrated good potential to solve the problem. Nevertheless, most of the existing works of NAS focus on the search algorithms and pay little attention to the search space. In this paper, we argue that the search space is also an important human prior to the success of NAS in different applications. Thus, we propose a novel search space tailored for text representation. Through automatic search, the discovered network architecture outperforms state-of-the-art models on various public datasets on text classification and natural language inference tasks. Furthermore, some of the design principles found in the automatic network agree well with human intuition.