mlsanityData to 7 Oct 2026

BERTs are Generative In-Context Learners

arXiv 2406.04823 ↗First posted 7 Jun 2024cs.CL cs.AIcited by 11

Not on the map. Why?

Abstract (arXiv)

While in-context learning is commonly associated with causal language models, such as GPT, we demonstrate that this capability also 'emerges' in masked language models. Through an embarrassingly simple inference technique, we enable an existing masked model, DeBERTa, to perform generative tasks without additional training or architectural changes. Our evaluation reveals that the masked and causal language models behave very differently, as they clearly outperform each other on different categories of tasks. These complementary strengths suggest that the field's focus on causal models for in-context learning may be limiting - both architectures can develop these capabilities, but with distinct advantages; pointing toward promising hybrid approaches that combine the strengths of both objectives.

Key facts. Cited by 11 papers in the map (data through 7 Oct 2026). Builds on 42 of its 62 references that are in the map. First posted on arXiv 7 Jun 2024.

Builds on

42 of 62 references.

References · 42

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  5. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  6. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  7. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  8. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  9. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,895
  10. 2019The Curious Case of Neural Text Degenerationcited by 2,881
  11. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
  12. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
30 more
  1. 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,157
  2. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,140
  3. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,114
  4. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  5. 2018A Call for Clarity in Reporting BLEU Scorescited by 1,790
  6. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,718
  7. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,270
  8. 2020Making Pre-trained Language Models Better Few-shot Learnerscited by 1,255
  9. 2015Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Bookscited by 1,041
  10. 2021GLM: General Language Model Pretraining with Autoregressive Blank Infillingcited by 1,026
  11. 2016The LAMBADA dataset: Word prediction requiring a broad discourse contextcited by 960
  12. 2022In-context Learning and Induction Headscited by 930
  13. 2022Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learningcited by 886
  14. 2019Unified Language Model Pre-training for Natural Language Understanding and Generationcited by 668
  15. 2020It's Not Just Size That Matters: Small Language Models Are Also Few-Shot Learnerscited by 668
  16. 2016A Corpus and Evaluation Framework for Deeper Understanding of Commonsense Storiescited by 413
  17. 2018WiC: the Word-in-Context Dataset for Evaluating Context-Sensitive Meaning Representationscited by 370
  18. 2021Few-shot Learning with Multilingual Language Modelscited by 278
  19. 2018A Simple Method for Commonsense Reasoningcited by 223
  20. 2022Demystifying Prompts in Language Models via Perplexity Estimationcited by 215
  21. 2018ReCoRD: Bridging the Gap between Human and Machine Commonsense Reading Comprehensioncited by 198
  22. 2019BERT has a Mouth, and It Must Speak: BERT as a Markov Random Field Language Modelcited by 184
  23. 2023Large Language Models Are Latent Variable Models: Explaining and Finding Good Demonstrations for In-Context Learningcited by 149
  24. 2022CM3: A Causal Masked Multimodal Model of the Internetcited by 142
  25. 2023The unreasonable effectiveness of few-shot learning for machine translationcited by 91
  26. 2021Larger-Scale Transformers for Multilingual Masked Language Modelingcited by 86
  27. 2020A Mathematical Exploration of Why Language Models Help Solve Downstream Taskscited by 71
  28. 2023A Better Way to Do Masked Language Model Scoringcited by 27
  29. 2023CausalLM is not optimal for in-context learningcited by 26
  30. 2024More Room for Language: Investigating the Effect of Retrieval on Language Modelscited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 4; 2025: 5; 2026: 2.

11 papers.

  1. 2024Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder for Fast, Memory Efficient, and Long Context Finetuning and Inferencecited by 527
  2. 2025Seq vs Seq: An Open Suite of Paired Encoders and Decoderscited by 34
  3. 2024GPT or BERT: why not both?cited by 26
  4. 2024Genomic Language Models: Opportunities and Challengescited by 20
  5. 2024Small Languages, Big Models: A Study of Continual Training on Languages of Norwaycited by 12
  6. 2025The Atlas of In-Context Learning: How Attention Heads Shape In-Context Retrieval Augmentationcited by 11
  7. 2025Enabling Autoregressive Models to Fill In Masked Tokenscited by 7
  8. 2025TEN: Table Explicitization, Neurosymbolicallycited by 0
  9. 2025Dual-objective Language Models: Training Efficiency Without Overfittingcited by 0
  10. 2026A Unified Framework for In-Context Learning with Causal and Masked Language Modelscited by 0
  11. 2026Subword Segmental BabyLMs: Learning to Tokenise for Sample-Efficient Pretrainingcited by 0