mlsanityData to 14 Oct 2026

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

arXiv 1810.04805 ↗First posted 11 Oct 2018cs.CLcited by 40,399

Not on the map. Why?

Cited by 901 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We introduce a new language representation model called BERT, which stands for Bidirectional Encoder Representations from Transformers. Unlike recent language representation models, BERT is designed to pre-train deep bidirectional representations from unlabeled text by jointly conditioning on both left and right context in all layers. As a result, the pre-trained BERT model can be fine-tuned with just one additional output layer to create state-of-the-art models for a wide range of tasks, such as question answering and language inference, without substantial task-specific architecture modifications. BERT is conceptually simple and empirically powerful. It obtains new state-of-the-art results on eleven natural language processing tasks, including pushing the GLUE score to 80.5% (7.7% point absolute improvement), MultiNLI accuracy to 86.7% (4.6% absolute improvement), SQuAD v1.1 question answering Test F1 to 93.2 (1.5 point absolute improvement) and SQuAD v2.0 Test F1 to 83.1 (5.1 point absolute improvement).

Key facts. Cited by 40,399 papers in the map (data through 14 Oct 2026). First posted on arXiv 11 Oct 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2019: 4; 2020: 11; 2021: 6421; 2022: 7471; 2023: 8933; 2024: 8162; 2025: 6099; 2026: 3298.

40,399 papers.

Sort:

By area:

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  3. 2023GPT-4 Technical Reportcited by 18,396
  4. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  6. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  7. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  8. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  9. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,974
  10. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
  11. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,968
  12. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,355
38 more
  1. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  2. 2022Scalable Diffusion Models with Transformerscited by 6,054
  3. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  4. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  5. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  6. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  7. 2022Large Language Models are Zero-Shot Reasonerscited by 4,374
  8. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,322
  9. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,182
  10. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,915
  11. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,864
  12. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,795
  13. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
  14. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  15. 2021Program Synthesis with Large Language Modelscited by 3,637
  16. 2022A ConvNet for the 2020scited by 3,579
  17. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  18. 2023Qwen Technical Reportcited by 3,344
  19. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,332
  20. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
  21. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,711
  22. 2022Locating and Editing Factual Associations in GPTcited by 2,691
  23. 2023A Survey of Large Language Modelscited by 2,576
  24. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,565
  25. 2022Emergent Abilities of Large Language Modelscited by 2,512
  26. 2021CLIPScore: A Reference-free Evaluation Metric for Image Captioningcited by 2,424
  27. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,375
  28. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,348
  29. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,342
  30. 2022Video Diffusion Modelscited by 2,305
  31. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,276
  32. 2022An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversioncited by 2,245
  33. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,234
  34. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
  35. 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,173
  36. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,141
  37. 2022Survey of Hallucination in Natural Language Generationcited by 2,075
  38. 2022A Time Series is Worth 64 Words: Long-term Forecasting with Transformerscited by 2,039

50 most cited of 40,399 All 40,399 (JSON)