mlsanityData to 14 Oct 2026

Not all layers are equally as important: Every Layer Counts BERT

arXiv 2311.02265 ↗First posted 3 Nov 2023cs.CLcited by 21

Not on the map. Why?

Abstract (arXiv)

This paper introduces a novel modification of the transformer architecture, tailored for the data-efficient pretraining of language models. This aspect is evaluated by participating in the BabyLM challenge, where our solution won both the strict and strict-small tracks. Our approach allows each transformer layer to select which outputs of previous layers to process. The empirical results verify the potential of this simple modification and show that not all layers are equally as important.

Key facts. Cited by 21 papers in the map (data through 14 Oct 2026). Builds on 23 of its 46 references that are in the map. First posted on arXiv 3 Nov 2023.

Builds on

23 of 46 references.

References · 23

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  4. 2020Language Models are Few-Shot Learnerscited by 32,419
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  6. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  7. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,148
  8. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,682
  9. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,183
  10. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,729
  11. 2019Energy and Policy Considerations for Deep Learning in NLPcited by 1,254
  12. 2019BERT Rediscovers the Classical NLP Pipelinecited by 1,059
11 more
  1. 2018Neural Network Acceptability Judgmentscited by 958
  2. 2020The State and Fate of Linguistic Diversity and Inclusion in the NLP Worldcited by 798
  3. 2019BLiMP: The Benchmark of Linguistic Minimal Pairs for Englishcited by 564
  4. 2020ReZero is All You Need: Fast Convergence at Large Depthcited by 264
  5. 2020The Computational Limits of Deep Learningcited by 193
  6. 2015Training Very Deep Networkscited by 173
  7. 2015The Goldilocks Principle: Reading Children's Books with Explicit Memory Representationscited by 108
  8. 201975 Languages, 1 Model: Parsing Universal Dependencies Universallycited by 90
  9. 2018A standardized Project Gutenberg corpus for statistical analysis of natural language and quantitative linguisticscited by 64
  10. 2023Trained on 100 million words and still in shape: BERT meets British National Corpuscited by 39
  11. 2017An optimal transportation approach for assessing almost stochastic ordercited by 33

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 7; 2025: 9; 2026: 5.

21 papers.

Sort:

By area:

  1. 2024LISA: Layerwise Importance Sampling for Memory-Efficient Large Language Model Fine-Tuningcited by 89
  2. 2025Findings of the BabyLM Challenge: Sample-Efficient Pretraining on Developmentally Plausible Corporacited by 88
  3. 2024DenseFormer: Enhancing Information Flow in Transformers via Depth Weighted Averagingcited by 53
  4. 2024Findings of the Second BabyLM Challenge: Sample-Efficient Pretraining on Developmentally Plausible Corporacited by 48
  5. 2024GPT or BERT: why not both?cited by 27
  6. 2024From Babble to Words: Pre-Training Language Models on Continuous Streams of Phonemescited by 11
  7. 2025Dialogue Is Not Enough to Make a Communicative BabyLM (But Neither Is Developmentally Inspired Reinforcement Learning)cited by 11
  8. 2025BabyBabelLM: A Multilingual Benchmark of Developmentally Plausible Training Datacited by 10
  9. 2025IPA-CHILDES & G2P+: Feature-Rich Resources for Cross-Lingual Phonology and Phonemic Language Modelingcited by 6
  10. 2025Towards Developmentally Plausible Rewards: Communicative Success as a Learning Signal for Interactive Language Modelscited by 4
  11. 2025Unpacking Let Alone: Human-Scale Models Generalize to a Rare Construction in Form but not Meaningcited by 4
  12. 2026Residual Stream Duality in Modern Transformer Architecturescited by 3
9 more
  1. 2025Masked Diffusion Language Models with Frequency-Informed Trainingcited by 2
  2. 2025Are Multilingual Language Models an Off-ramp for Under-resourced Languages? Will we arrive at Digital Language Equality in Europe in 2030?cited by 1
  3. 2026The Grammar of Transformers: A Systematic Review of Interpretability Research on Syntactic Knowledge in Language Modelscited by 1
  4. 2026Learning from Child-Directed Speech in Two-Language Scenarios: A French-English Case Studycited by 1
  5. 2024Team Ryu's Submission to SIGMORPHON 2024 Shared Task on Subword Tokenizationcited by 0
  6. 2024A surprisal oracle for when every layer countscited by 0
  7. 2025BabyLMs for isiXhosa: Data-Efficient Language Modelling in a Low-Resource Contextcited by 0
  8. 2026Right Tool, Right Job: Native-Language Evaluation, Tokenizer Sensitivity, and Methodological Findings from a French-Only BabyLMcited by 0
  9. 2026A retrospective analysis on the use of LLMs to study infant syntax learningcited by 0