# AutoTrans: Automating Transformer Design via Reinforced Architecture Search

arXiv: https://arxiv.org/abs/2009.02070 · HTML: https://mlsanity.com/paper/2009.02070/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 4 Sep 2020.

## Built on by (most cited first, 12 of 12)

- [GLiT: Neural Architecture Search for Global and Local Image Transformer](https://mlsanity.com/paper/2107.02960/) (2021)
- [ALoRA: Allocating Low-Rank Adaptation for Fine-tuning Large Language Models](https://mlsanity.com/paper/2403.16187/) (2024)
- [A Survey of Techniques for Optimizing Transformer Inference](https://mlsanity.com/paper/2307.07982/) (2023)
- [MiLoRA: Efficient Mixture of Low-Rank Adaptation for Large Language Models Fine-tuning](https://mlsanity.com/paper/2410.18035/) (2024)
- [Overview of the PromptCBLUE Shared Task in CHIP2023](https://mlsanity.com/paper/2312.17522/) (2023)
- [Candidate Soups: Fusing Candidate Results Improves Translation Quality for Non-Autoregressive Translation](https://mlsanity.com/paper/2301.11503/) (2023)
- [Text2MDT: Extracting Medical Decision Trees from Medical Texts](https://mlsanity.com/paper/2401.02034/) (2024)
- [IAPT: Instruction-Aware Prompt Tuning for Large Language Models](https://mlsanity.com/paper/2405.18203/) (2024)
- [Evaluating ChatGPT on Medical Information Extraction Tasks: Performance, Explainability and Beyond](https://mlsanity.com/paper/2601.21767/) (2026)
- [Automatic Pharma News Categorization](https://mlsanity.com/paper/2201.00688/) (2021)
- [Aligning Programming Language and Natural Language: Exploring Design Choices in Multi-Modal Transformer-Based Embedding for Bug Localization](https://mlsanity.com/paper/2406.17615/) (2024)
- [A Generative Model for Joint Multiple Intent Detection and Slot Filling](https://mlsanity.com/paper/2602.08322/) (2026)

## Abstract (arXiv)

Though the transformer architectures have shown dominance in many natural language understanding tasks, there are still unsolved issues for the training of transformer models, especially the need for a principled way of warm-up which has shown importance for stable training of a transformer, as well as whether the task at hand prefer to scale the attention product or not. In this paper, we empirically explore automating the design choices in the transformer model, i.e., how to set layer-norm, whether to scale, number of layers, number of heads, activation function, etc, so that one can obtain a transformer architecture that better suits the tasks at hand. RL is employed to navigate along search space, and special parameter sharing strategies are designed to accelerate the search. It is shown that sampling a proportion of training data per epoch during search help to improve the search quality. Experiments on the CoNLL03, Multi-30k, IWSLT14 and WMT-14 shows that the searched transformer model can outperform the standard transformers. In particular, we show that our learned model can be trained more robustly with large learning rates without warm-up.