mlsanityData to 7 Oct 2026

ExtremeBERT: A Toolkit for Accelerating Pretraining of Customized BERT

arXiv 2211.17201 ↗First posted 30 Nov 2022cs.CL cs.LG math.OCcited by 7

Not on the map. Why?

Abstract (arXiv)

In this paper, we present ExtremeBERT, a toolkit for accelerating and customizing BERT pretraining. Our goal is to provide an easy-to-use BERT pretraining toolkit for the research community and industry. Thus, the pretraining of popular language models on customized datasets is affordable with limited resources. Experiments show that, to achieve the same or better GLUE scores, the time cost of our toolkit is over 6× times less for BERT Base and 9× times less for BERT Large when compared with the original BERT paper. The documentation and code are released at https://github.com/extreme-bert/extreme-bert under the Apache-2.0 license.

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Builds on 8 of its 11 references that are in the map. First posted on arXiv 30 Nov 2022.

Builds on

8 of 11 references.

References · 8

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  5. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,718
  6. 2018Sentence Encoders on STILTs: Supplementary Training on Intermediate Labeled-data Taskscited by 214
  7. 2021How to Train BERT with an Academic Budgetcited by 80
  8. 2021Eigencurve: Optimal Learning Rate Schedule for SGD on Quadratic Objectives with Skewed Hessian Spectrumscited by 15

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 3; 2024: 1; 2025: 2; 2026: 1.

7 papers.

  1. 2023On Efficient Training of Large-Scale Deep Learning Models: A Literature Reviewcited by 26
  2. 2024AdaGrad under Anisotropic Smoothnesscited by 22
  3. 2023Accelerated Convergence of Stochastic Heavy Ball Method under Anisotropic Gradient Noisecited by 9
  4. 2023Data-Efficient French Language Modeling with CamemBERTacited by 8
  5. 2025Unbiased Gradient Low-Rank Projectioncited by 6
  6. 2025Stepsize anything: A unified learning rate schedule for budgeted-iteration trainingcited by 1
  7. 2026Translation as a Computationally Efficient Bridge: Feasibility of English BERT for Low-Resource Languagescited by 0