mlsanityData to 14 Oct 2026

Memory-Efficient Differentiable Transformer Architecture Search

arXiv 2105.14669 ↗First posted 31 May 2021cs.LG cs.CLcited by 13

Not on the map. Why?

Abstract (arXiv)

Differentiable architecture search (DARTS) is successfully applied in many vision tasks. However, directly using DARTS for Transformers is memory-intensive, which renders the search process infeasible. To this end, we propose a multi-split reversible network and combine it with DARTS. Specifically, we devise a backpropagation-with-reconstruction algorithm so that we only need to store the last layer's outputs. By relieving the memory burden for DARTS, it allows us to search with larger hidden size and more candidate operations. We evaluate the searched architecture on three sequence-to-sequence datasets, i.e., WMT'14 English-German, WMT'14 English-French, and WMT'14 English-Czech. Experimental results show that our network consistently outperforms standard Transformers across the tasks. Moreover, our method compares favorably with big-size Evolved Transformers, reducing search computation by an order of magnitude.

Key facts. Cited by 13 papers in the map (data through 14 Oct 2026). Builds on 46 of its 51 references that are in the map. First posted on arXiv 31 May 2021.

Builds on

46 of 51 references.

References · 46

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  4. 2016Layer Normalizationcited by 5,669
  5. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,918
  6. 2018DARTS: Differentiable Architecture Searchcited by 1,798
  7. 2020Reformer: The Efficient Transformercited by 1,668
  8. 2018Glow: Generative Flow with Invertible 1x1 Convolutionscited by 1,545
  9. 2016Neural Architecture Search with Reinforcement Learningcited by 1,499
  10. 2019Energy and Policy Considerations for Deep Learning in NLPcited by 1,254
  11. 2017Learning Transferable Architectures for Scalable Image Recognitioncited by 1,101
  12. 2018Regularized Evolution for Image Classifier Architecture Searchcited by 909
34 more
  1. 2018Efficient Neural Architecture Search via Parameter Sharingcited by 837
  2. 2015Improving Neural Machine Translation Models with Monolingual Datacited by 788
  3. 2019Once-for-All: Train One Network and Specialize it for Efficient Deploymentcited by 787
  4. 2016Language Modeling with Gated Convolutional Networkscited by 702
  5. 2018ProxylessNAS: Direct Neural Architecture Search on Target Task and Hardwarecited by 673
  6. 2019Deep Equilibrium Modelscited by 615
  7. 2019Single Path One-Shot Neural Architecture Search with Uniform Samplingcited by 414
  8. 2018Invertible Residual Networkscited by 365
  9. 2017Large-Scale Evolution of Image Classifierscited by 348
  10. 2018SNAS: Stochastic Neural Architecture Searchcited by 289
  11. 2016Designing Neural Network Architectures using Reinforcement Learningcited by 279
  12. 2019PC-DARTS: Partial Channel Connections for Memory-Efficient Architecture Searchcited by 265
  13. 2019Progressive Differentiable Architecture Search: Bridging the Depth Gap between Search and Evaluationcited by 230
  14. 2017The Reversible Residual Network: Backpropagation Without Storing Activationscited by 217
  15. 2019Pay Less Attention with Lightweight and Dynamic Convolutionscited by 211
  16. 2018Neural Autoregressive Flowscited by 208
  17. 2017SMASH: One-Shot Model Architecture Search through HyperNetworkscited by 202
  18. 2017Hierarchical Representations for Efficient Architecture Searchcited by 191
  19. 2019The Evolved Transformercited by 180
  20. 2020Multiscale Deep Equilibrium Modelscited by 151
  21. 2018i-RevNet: Deep Invertible Networkscited by 137
  22. 2020HAT: Hardware-Aware Transformers for Efficient Natural Language Processingcited by 136
  23. 2017Evolving Deep Neural Networkscited by 95
  24. 2019FlowSeq: Non-Autoregressive Conditional Sequence Generation with Generative Flowcited by 89
  25. 2017Reversible Architectures for Arbitrarily Deep Residual Neural Networkscited by 70
  26. 2020DSNAS: Direct Neural Architecture Search without Parameter Retrainingcited by 69
  27. 2019Discrete Flows: Invertible Generative Models of Discrete Datacited by 68
  28. 2020Neural Architecture Transfercited by 60
  29. 2018Path-Level Network Transformation for Efficient Architecture Searchcited by 44
  30. 2020CoDA: Contrast-enhanced and Diversity-promoting Data Augmentation for Natural Language Understandingcited by 41
  31. 2019AtomNAS: Fine-Grained End-to-End Neural Architecture Searchcited by 40
  32. 2020ISTA-NAS: Efficient and Consistent Neural Architecture Search by Sparse Codingcited by 36
  33. 2019Efficient Neural Architecture Search via Proximal Iterationscited by 31
  34. 2018Reversible Recurrent Neural Networkscited by 23

Built on by

Cited by year

Citing papers per year: 2021: 3; 2022: 5; 2023: 2; 2024: 1; 2025: 2.

13 papers.

Sort:

  1. 2021A Survey of Transformerscited by 401
  2. 2024Model Compression and Efficient Inference for Large Language Models: A Surveycited by 64
  3. 2023A Survey of Techniques for Optimizing Transformer Inferencecited by 35
  4. 2022Neural Architecture Search on Efficient Transformers and Beyondcited by 17
  5. 2021Multi-head or Single-head? An Empirical Comparison for Transformer Trainingcited by 16
  6. 2022LiteTransformerSearch: Training-free Neural Architecture Search for Efficient Language Modelscited by 14
  7. 2022Meta-Learning the Difference: Preparing Large Language Models for Efficient Adaptationcited by 14
  8. 2021RankNAS: Efficient Neural Architecture Search by Pairwise Rankingcited by 12
  9. 2023Training-free Neural Architecture Search for RNNs and Transformerscited by 9
  10. 2022RevBiFPN: The Fully Reversible Bidirectional Feature Pyramid Networkcited by 6
  11. 2022FocusFormer: Focusing on What We Need via Architecture Samplercited by 5
  12. 2025Universal Neural Architecture Space: Covering ConvNets, Transformers and Everything in Betweencited by 1
1 more
  1. 2025DASViT: Differentiable Architecture Search for Vision Transformercited by 0