mlsanityData to 7 Oct 2026

Which *BERT? A Survey Organizing Contextualized Encoders

arXiv 2010.00854 ↗First posted 2 Oct 2020cs.CL cs.LGcited by 26

Not on the map. Why?

Abstract (arXiv)

Pretrained contextualized text encoders are now a staple of the NLP community. We present a survey on language representation learning with the aim of consolidating a series of shared lessons learned across a variety of recent efforts. While significant advancements continue at a rapid pace, we find that enough has now been discovered, in different directions, that we can begin to organize advances according to common themes. Through this organization, we highlight important considerations when interpreting recent contributions and choosing which model to use.

Key facts. Cited by 26 papers in the map (data through 7 Oct 2026). First posted on arXiv 2 Oct 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 9; 2022: 10; 2023: 4; 2024: 0; 2025: 1; 2026: 2.

26 papers.

Sort:

By area:

  1. 2023A Survey on Model Compression for Large Language Modelscited by 263
  2. 2022Self-Supervised Speech Representation Learning: A Reviewcited by 258
  3. 2022Is Neural Topic Modelling Better than Clustering? An Empirical Study on Clustering with Contextual Embeddings for Topicscited by 35
  4. 2021The Surprising Performance of Simple Baselines for Misinformation Detectioncited by 27
  5. 2023Gender Bias in BERT -- Measuring and Analysing Biases through Sentiment Rating in a Realistic Downstream Classification Taskcited by 21
  6. 2022Textual Manifold-based Defense Against Natural Language Adversarial Examplescited by 19
  7. 2023A Survey on Stability of Learning with Limited Labelled Data and its Sensitivity to the Effects of Randomnesscited by 14
  8. 2021SlovakBERT: Slovak Masked Language Modelcited by 11
  9. 2022Textwash -- automated open-source text anonymisationcited by 10
  10. 2021Fine-tuning Encoders for Improved Monolingual and Zero-shot Polylingual Neural Topic Modelingcited by 8
  11. 2021Optimizing Inference Performance of Transformers on CPUscited by 7
  12. 2021FHAC at GermEval 2021: Identifying German toxic, engaging, and fact-claiming comments with ensemble learningcited by 7
14 more
  1. 2021Deep Transfer Learning & Beyond: Transformer Language Models in Information Systems Researchcited by 7
  2. 2022Evidence > Intuition: Transferability Estimation for Encoder Selectioncited by 7
  3. 2022An Assessment of the Impact of OCR Noise on Language Modelscited by 5
  4. 2022Empirical Evaluation and Theoretical Analysis for Representation Learning: A Surveycited by 5
  5. 2022MiniALBERT: Model Distillation via Parameter-Efficient Recursive Transformerscited by 5
  6. 2023A Predictive Factor Analysis of Social Biases and Task-Performance in Pretrained Masked Language Modelscited by 4
  7. 2026STRUCTSURVEY: Structured Agentic Retrieval for Automated Survey Paper Generationcited by 3
  8. 2021A Semi-supervised Multi-task Learning Approach to Classify Customer Contact Intentscited by 1
  9. 2022Fusing Sentence Embeddings Into LSTM-based Autoregressive Language Modelscited by 1
  10. 2021FH-SWF SG at GermEval 2021: Using Transformer-Based Language Models to Identify Toxic, Engaging, & Fact-Claiming Commentscited by 0
  11. 2021ur-iw-hnt at GermEval 2021: An Ensembling Strategy with Multiple BERT Modelscited by 0
  12. 2022An Empirical Study on Learning and Improving the Search Objective for Unsupervised Paraphrasingcited by 0
  13. 2025Polarity Detection of Sustainable Development Goals in News Textcited by 0
  14. 2026Self-Supervised Learning for Speaker Recognition: A study and reviewcited by 0