mlsanityData to 7 Oct 2026

RoBERTa: A Robustly Optimized BERT Pretraining Approach

arXiv 1907.11692 ↗First posted 26 Jul 2019cs.CLcited by 13,266

Not on the map. Why?

Cited by 446 in LLM reasoning, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Language model pretraining has led to significant performance gains but careful comparison between different approaches is challenging. Training is computationally expensive, often done on private datasets of different sizes, and, as we will show, hyperparameter choices have significant impact on the final results. We present a replication study of BERT pretraining (Devlin et al., 2019) that carefully measures the impact of many key hyperparameters and training data size. We find that BERT was significantly undertrained, and can match or exceed the performance of every model published after it. Our best model achieves state-of-the-art results on GLUE, RACE and SQuAD. These results highlight the importance of previously overlooked design choices, and raise questions about the source of recently reported improvements. We release our models and code.

Key facts. Cited by 13,266 papers in the map (data through 7 Oct 2026). First posted on arXiv 26 Jul 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2020: 3; 2021: 2051; 2022: 2552; 2023: 3045; 2024: 2736; 2025: 1838; 2026: 1041.

13,266 papers.

Sort:

By area:

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  5. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  6. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  7. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  8. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  9. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  10. 2023Qwen Technical Reportcited by 3,273
  11. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  12. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,748
38 more
  1. 2023A Survey of Large Language Modelscited by 2,537
  2. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,534
  3. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,360
  4. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
  5. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,222
  6. 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,168
  7. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,126
  8. 2023Code Llama: Open Foundation Models for Codecited by 2,105
  9. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
  10. 2021Vision Transformers for Dense Predictioncited by 1,789
  11. 2022Make-A-Video: Text-to-Video Generation without Text-Video Datacited by 1,725
  12. 2021Restormer: Efficient Transformer for High-Resolution Image Restorationcited by 1,618
  13. 2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?cited by 1,413
  14. 2021BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-modelscited by 1,234
  15. 2021BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Modelscited by 1,225
  16. 2021DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharingcited by 1,219
  17. 2022LaMDA: Language Models for Dialog Applicationscited by 1,218
  18. 2022Text Embeddings by Weakly-Supervised Contrastive Pre-trainingcited by 1,216
  19. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,209
  20. 2021What Makes Good In-Context Examples for GPT-3?cited by 1,184
  21. 2021SimMIM: A Simple Framework for Masked Image Modelingcited by 1,147
  22. 2021Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivitycited by 1,122
  23. 2021Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNetcited by 1,096
  24. 2021Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationcited by 1,085
  25. 2021MUSIQ: Multi-scale Image Quality Transformercited by 1,058
  26. 2022Inner Monologue: Embodied Reasoning through Planning with Language Modelscited by 1,057
  27. 2022Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agentscited by 1,029
  28. 2021GLM: General Language Model Pretraining with Autoregressive Blank Infillingcited by 1,026
  29. 2021Transformers in Vision: A Surveycited by 1,009
  30. 2024Tulu 3: Pushing Frontiers in Open Language Model Post-Trainingcited by 988
  31. 2021Ethical and social risks of harm from Language Modelscited by 957
  32. 2021An Explanation of In-context Learning as Implicit Bayesian Inferencecited by 915
  33. 2021MuSiQue: Multihop Questions via Single-hop Question Compositioncited by 893
  34. 2021CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generationcited by 890
  35. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Expertscited by 890
  36. 2022LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scalecited by 881
  37. 2021Towards a Unified View of Parameter-Efficient Transfer Learningcited by 863
  38. 2021Are NLP Models really able to Solve Simple Math Word Problems?cited by 861

50 most cited of 13,266 All 13,266 (JSON)