mlsanityData to 7 Oct 2026

GPT or BERT: why not both?

arXiv 2410.24159 ↗First posted 31 Oct 2024cs.CLcited by 26

Not on the map. Why?

Abstract (arXiv)

We present a simple way to merge masked language modeling with causal language modeling. This hybrid training objective results in a model that combines the strengths of both modeling paradigms within a single transformer stack: GPT-BERT can be transparently used like any standard causal or masked language model. We test the pretraining process that enables this flexible behavior on the BabyLM Challenge 2024. The results show that the hybrid pretraining outperforms masked-only or causal-only models. We openly release the models, training corpora and code.

Key facts. Cited by 26 papers in the map (data through 7 Oct 2026). Builds on 30 of its 70 references that are in the map. First posted on arXiv 31 Oct 2024.

Builds on

30 of 70 references.

References · 30

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  5. 2017On Calibration of Modern Neural Networkscited by 4,860
  6. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  7. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  8. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
  9. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,140
  10. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,718
  11. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,199
  12. 2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Modelcited by 1,093
18 more
  1. 2021GLM: General Language Model Pretraining with Autoregressive Blank Infillingcited by 1,026
  2. 2016The LAMBADA dataset: Word prediction requiring a broad discourse contextcited by 960
  3. 2018Neural Network Acceptability Judgmentscited by 952
  4. 2021Probing Classifiers: Promises, Shortcomings, and Advancescited by 831
  5. 2019Unified Language Model Pre-training for Natural Language Understanding and Generationcited by 668
  6. 2019BLiMP: The Benchmark of Linguistic Minimal Pairs for Englishcited by 553
  7. 2024LLM2Vec: Large Language Models Are Secretly Powerful Text Encoderscited by 396
  8. 2019BERT has a Mouth, and It Must Speak: BERT as a Markov Random Field Language Modelcited by 184
  9. 2022CM3: A Causal Masked Multimodal Model of the Internetcited by 142
  10. 2023Baby Llama: knowledge distillation from an ensemble of teachers trained on a small dataset with no performance penaltycited by 57
  11. 2024Elements of World Knowledge (EWoK): A Cognition-Inspired Framework for Evaluating Basic World Knowledge in Language Modelscited by 55
  12. 2024DenseFormer: Enhancing Information Flow in Transformers via Depth Weighted Averagingcited by 48
  13. 2023Trained on 100 million words and still in shape: BERT meets British National Corpuscited by 39
  14. 2024[Call for Papers] The 2nd BabyLM Challenge: Sample-efficient pretraining on a developmentally plausible corpuscited by 26
  15. 2023Not all layers are equally as important: Every Layer Counts BERTcited by 21
  16. 2023Dynamic Masking Rate Schedules for MLM Pretrainingcited by 11
  17. 2024BERTs are Generative In-Context Learnerscited by 11
  18. 2024ENTP: Encoder-only Next Token Predictioncited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 12; 2026: 12.

26 papers.

Sort:

By area:

  1. 2025Seq vs Seq: An Open Suite of Paired Encoders and Decoderscited by 34
  2. 2025BabyLM Turns 3: Call for papers for the 2025 BabyLM workshopcited by 24
  3. 2025Should We Still Pretrain Encoders with Masked Language Modeling?cited by 15
  4. 2024Small Languages, Big Models: A Study of Continual Training on Languages of Norwaycited by 12
  5. 2026BabyLM Turns 4 and Goes Multilingual: Call for Papers for the 2026 BabyLM Workshopcited by 10
  6. 2025BabyBabelLM: A Multilingual Benchmark of Developmentally Plausible Training Datacited by 9
  7. 2025Teacher Demonstrations in a BabyLM's Zone of Proximal Development for Contingent Multi-Turn Interactioncited by 9
  8. 2025IPA-CHILDES & G2P+: Feature-Rich Resources for Cross-Lingual Phonology and Phonemic Language Modelingcited by 6
  9. 2026Bringing Up a Bilingual BabyLM: Investigating Multilingual Language Acquisition Using Small-Scale Modelscited by 4
  10. 2025LongTail-Swap: benchmarking language models' abilities on rare wordscited by 3
  11. 2025Enhancing next token prediction based pre-training for jet foundation modelscited by 3
  12. 2025Masked Diffusion Language Models with Frequency-Informed Trainingcited by 2
14 more
  1. 2026DHPLT: large-scale multilingual diachronic corpora and word representations for semantic change modellingcited by 1
  2. 2026Perturbation: A simple and efficient adversarial tracer for representation learning in language modelscited by 1
  3. 2024ELECTRA and GPT-4o: Cost-Effective Partners for Sentiment Analysiscited by 0
  4. 2025BabyLM's First Constructions: Causal probing provides a signal of learningcited by 0
  5. 2025Decoding Partial Differential Equations: Cross-Modal Adaptation of Decoder-only Models to PDEscited by 0
  6. 2025Dual-objective Language Models: Training Efficiency Without Overfittingcited by 0
  7. 2026Bias Dynamics in BabyLMs: Towards a Compute-Efficient Sandbox for Democratising Pre-Training Debiasingcited by 0
  8. 2026CxMP: A Linguistic Minimal-Pair Benchmark for Evaluating Constructional Understanding in Language Modelscited by 0
  9. 2026The First ChineseBabyLM Challenge: training data-efficient and cognitively plausible language models for Chinesecited by 0
  10. 2026Squeezing More from Limited Data with Recursive Transformerscited by 0
  11. 2026Looped GPT-BERT: Trading Parameters for Computation in Small Language Modelingcited by 0
  12. 2026Right Tool, Right Job: Native-Language Evaluation, Tokenizer Sensitivity, and Methodological Findings from a French-Only BabyLMcited by 0
  13. 2026Relational Attention for Data-Efficient Language Modelingcited by 0
  14. 2026A retrospective analysis on the use of LLMs to study infant syntax learningcited by 0