Neural Architecture Search for Sentence Classification with BERT
Not on the map. Why?
Abstract (arXiv)
Pre training of language models on large text corpora is common practice in Natural Language Processing. Following, fine tuning of these models is performed to achieve the best results on a variety of tasks. In this paper we question the common practice of only adding a single output layer as a classification head on top of the network. We perform an AutoML search to find architectures that outperform the current single layer at only a small compute cost. We validate our classification architecture on a variety of NLP benchmarks from the GLUE dataset.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 10 of its 14 references that are in the map. First posted on arXiv 27 Mar 2024.
Builds on
10 of 14 references.
References · 10
- 2017Attention Is All You Needcited by 60,361
- 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
- 2020Language Models are Few-Shot Learnerscited by 32,419
- 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
- 2019Parameter-Efficient Transfer Learning for NLPcited by 4,236
- 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,218
- 2015A large annotated corpus for learning natural language inferencecited by 1,847
- 2016Hyperband: A Novel Bandit-Based Approach to Hyperparameter Optimizationcited by 699
- 2019AutoML: A Survey of the State-of-the-Artcited by 390
- 2019SMART: Robust and Efficient Fine-Tuning for Pre-trained Natural Language Models through Principled Regularized Optimizationcited by 254
Built on by
Not cited yet.