mlsanityData to 7 Oct 2026

What's Hidden in a One-layer Randomly Weighted Transformer?

arXiv 2109.03939 ↗First posted 8 Sep 2021cs.CL cs.AIcited by 3

Not on the map. Why?

Abstract (arXiv)

We demonstrate that, hidden within one-layer randomly weighted neural networks, there exist subnetworks that can achieve impressive performance, without ever modifying the weight initializations, on machine translation tasks. To find subnetworks for one-layer randomly weighted neural networks, we apply different binary masks to the same weight matrix to generate different layers. Hidden within a one-layer randomly weighted Transformer, we find that subnetworks that can achieve 29.45/17.29 BLEU on IWSLT14/WMT14. Using a fixed pre-trained embedding layer, the previously found subnetworks are smaller than, but can match 98%/92% (34.14/25.24 BLEU) of the performance of, a trained Transformer small/base on IWSLT14/WMT14. Furthermore, we demonstrate the effectiveness of larger and deeper transformers in this setting, as well as the impact of different initialization methods. We released the source code at https://github.com/sIncerass/one_layer_lottery_ticket.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 37 of its 57 references that are in the map. First posted on arXiv 8 Sep 2021.

Builds on

37 of 57 references.

References · 37

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  6. 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,346
  7. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
  8. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
  9. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,250
  10. 2013Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computationcited by 2,031
  11. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  12. 2019TinyBERT: Distilling BERT for Natural Language Understandingcited by 1,153
25 more
  1. 2019Are Sixteen Heads Really Better than One?cited by 803
  2. 2019Linear Mode Connectivity and the Lottery Ticket Hypothesiscited by 627
  3. 2018Universal Transformerscited by 613
  4. 2019Deep Equilibrium Modelscited by 603
  5. 2020MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devicescited by 476
  6. 2019Reducing Transformer Depth on Demand with Structured Dropoutcited by 434
  7. 2017Supervised Learning of Universal Sentence Representations from Natural Language Inference Datacited by 428
  8. 2020Movement Pruning: Adaptive Sparsity by Fine-Tuningcited by 398
  9. 2019Q-BERT: Hessian Based Ultra Low Precision Quantization of BERTcited by 355
  10. 2021Random Feature Attentioncited by 350
  11. 2020The Lottery Ticket Hypothesis for Pre-trained BERT Networkscited by 275
  12. 2020Comparing Rewinding and Fine-tuning in Neural Network Pruningcited by 238
  13. 2019What's Hidden in a Randomly Weighted Neural Network?cited by 236
  14. 2020Information-Theoretic Probing with Minimum Description Lengthcited by 230
  15. 2019Deconstructing Lottery Tickets: Zeros, Signs, and the Supermaskcited by 206
  16. 2018Scaling Neural Machine Translationcited by 204
  17. 2020When BERT Plays the Lottery, All Tickets Are Winningcited by 114
  18. 2019Playing the lottery with rewards and multiple languages: lottery tickets in RL and NLPcited by 73
  19. 2019Weight Agnostic Neural Networkscited by 71
  20. 2017Classical Structured Prediction Losses for Sequence to Sequence Learningcited by 55
  21. 2020Masked Language Modeling for Proteins via Linearly Scalable Long-Context Transformerscited by 49
  22. 2021Playing Lottery Tickets with Vision and Languagecited by 45
  23. 2019No Training Required: Exploring Random Encoders for Sentence Classificationcited by 30
  24. 2020Reservoir Transformerscited by 18
  25. 2020On the impressive performance of randomly weighted encoders in summarization taskscited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 2; 2026: 1.

3 papers.

  1. 2025Linguistic Blind Spots of Large Language Modelscited by 6
  2. 2025The Strong Lottery Ticket Hypothesis for Multi-Head Attention Mechanismscited by 2
  3. 2026Training-Free Universal Approximation by Prompting Random Transformerscited by 1