mlsanityData to 14 Oct 2026

Life after BERT: What do Other Muppets Understand about Language?

arXiv 2205.10696 ↗First posted 21 May 2022cs.CLcited by 4

Not on the map. Why?

Abstract (arXiv)

Existing pre-trained transformer analysis works usually focus only on one or two model families at a time, overlooking the variability of the architecture and pre-training objectives. In our work, we utilize the oLMpics benchmark and psycholinguistic probing datasets for a diverse set of 29 models including T5, BART, and ALBERT. Additionally, we adapt the oLMpics zero-shot setup for autoregressive models and evaluate GPT networks of different sizes. Our findings show that none of these models can resolve compositional questions in a zero-shot fashion, suggesting that this skill is not learnable using existing pre-training objectives. Furthermore, we find that global model decisions such as architecture, directionality, size of the dataset, and pre-training objective are not predictive of a model's linguistic capabilities.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Builds on 41 of its 62 references that are in the map. First posted on arXiv 21 May 2022.

Builds on

41 of 62 references.

References · 41

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  5. 2020Scaling Laws for Neural Language Modelscited by 6,586
  6. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,893
  7. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  8. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,307
  9. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,790
  10. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,411
  11. 2018Deep contextualized word representationscited by 2,290
  12. 2020GLU Variants Improve Transformercited by 1,804
29 more
  1. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,729
  2. 2019BERT Rediscovers the Classical NLP Pipelinecited by 1,059
  3. 2019What Does BERT Look At? An Analysis of BERT's Attentioncited by 1,003
  4. 2019Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inferencecited by 876
  5. 2017RACE: Large-scale ReAding Comprehension Dataset From Examinationscited by 861
  6. 2019Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Prunedcited by 856
  7. 2019Are Sixteen Heads Really Better than One?cited by 816
  8. 2019Large Batch Optimization for Deep Learning: Training BERT in 76 minutescited by 674
  9. 2019Unified Language Model Pre-training for Natural Language Understanding and Generationcited by 670
  10. 2018What you can cram into a single vector: Probing sentence embeddings for linguistic propertiescited by 457
  11. 2020How Can We Know When Language Models Know? On the Calibration of Language Models for Question Answeringcited by 445
  12. 2019Linguistic Knowledge and Transferability of Contextual Representationscited by 383
  13. 2019What BERT is not: Lessons from a new suite of psycholinguistic diagnostics for language modelscited by 320
  14. 2019Revealing the Dark Secrets of BERTcited by 301
  15. 2020Transformers as Soft Reasoners over Languagecited by 270
  16. 2018A Simple Method for Commonsense Reasoningcited by 223
  17. 2019Assessing BERT's Syntactic Abilitiescited by 192
  18. 2019oLMpics -- On what Language Model Pre-training Capturescited by 161
  19. 2020With Little Power Comes Great Responsibilitycited by 142
  20. 2020When BERT Plays the Lottery, All Tickets Are Winningcited by 114
  21. 2021BERT Busters: Outlier Dimensions that Disrupt Transformerscited by 112
  22. 2021Probing Across Time: What Does RoBERTa Know and When?cited by 73
  23. 2021Incorporating Residual and Normalization Layers into Analysis of Masked Language Modelscited by 51
  24. 2020On the Interplay Between Fine-tuning and Sentence-level Probing for Linguistic Knowledge in Pre-trained Transformerscited by 39
  25. 2021Fine-Tuned Transformers Show Clusters of Similar Representations Across Layerscited by 30
  26. 2021Exploring the Role of BERT Token Representations to Explain Sentence Probing Resultscited by 29
  27. 2020RICA: Evaluating Robust Inference Capabilities Based on Commonsense Axiomscited by 28
  28. 2020BERT Knows Punta Cana is not just beautiful, it's gorgeous: Ranking Scalar Adjectives with Contextualised Representationscited by 15
  29. 2021What's the best place for an AI conference, Vancouver or ______: Why completing comparative questions is difficultcited by 4

Built on by

Cited by year

Citing papers per year: 2023: 3; 2024: 1.

4 papers.

  1. 2023Language models are not naysayers: An analysis of language models on negation benchmarkscited by 74
  2. 2024Deconstructing In-Context Learning: Understanding Prompts via Corruptioncited by 4
  3. 2023Larger Probes Tell a Different Story: Extending Psycholinguistic Datasets Via In-Context Learningcited by 3
  4. 2023A Comparative Analysis of Pretrained Language Models for Text-to-Speechcited by 1