# Learning Architectures from an Extended Search Space for Language Modeling

arXiv: https://arxiv.org/abs/2005.02593 · HTML: https://mlsanity.com/paper/2005.02593/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 6 May 2020.

## Built on by (most cited first, 4 of 4)

- [Meta Learning for Natural Language Processing: A Survey](https://mlsanity.com/paper/2205.01500/) (2022)
- [Semeval-2022 Task 1: CODWOE -- Comparing Dictionaries and Word Embeddings](https://mlsanity.com/paper/2205.13858/) (2022)
- [RankNAS: Efficient Neural Architecture Search by Pairwise Ranking](https://mlsanity.com/paper/2109.07383/) (2021)
- [DORE: A Dataset For Portuguese Definition Generation](https://mlsanity.com/paper/2403.18018/) (2024)

## Abstract (arXiv)

Neural architecture search (NAS) has advanced significantly in recent years but most NAS systems restrict search to learning architectures of a recurrent or convolutional cell. In this paper, we extend the search space of NAS. In particular, we present a general approach to learn both intra-cell and inter-cell architectures (call it ESS). For a better search result, we design a joint learning method to perform intra-cell and inter-cell NAS simultaneously. We implement our model in a differentiable architecture search system. For recurrent neural language modeling, it outperforms a strong baseline significantly on the PTB and WikiText data, with a new state-of-the-art on PTB. Moreover, the learned architectures show good transferability to other systems. E.g., they improve state-of-the-art systems on the CoNLL and WNUT named entity recognition (NER) tasks and CoNLL chunking task, indicating a promising line of research on large-scale pre-learned architectures.