mlsanityData to 7 Oct 2026

ALBERT: A Lite BERT for Self-supervised Learning of Language Representations

arXiv 1909.11942 ↗First posted 26 Sep 2019cs.CL cs.AIcited by 2,772

Not on the map. Why?

Cited by 75 in LLM reasoning, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Increasing model size when pretraining natural language representations often results in improved performance on downstream tasks. However, at some point further model increases become harder due to GPU/TPU memory limitations and longer training times. To address these problems, we present two parameter-reduction techniques to lower memory consumption and increase the training speed of BERT. Comprehensive empirical evidence shows that our proposed methods lead to models that scale much better compared to the original BERT. We also use a self-supervised loss that focuses on modeling inter-sentence coherence, and show it consistently helps downstream tasks with multi-sentence inputs. As a result, our best model establishes new state-of-the-art results on the GLUE, RACE, and squad benchmarks while having fewer parameters compared to BERT-large. The code and the pretrained models are available at https://github.com/google-research/ALBERT.

Key facts. Cited by 2,772 papers in the map (data through 7 Oct 2026). First posted on arXiv 26 Sep 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2020: 1; 2021: 694; 2022: 608; 2023: 568; 2024: 450; 2025: 272; 2026: 179.

2,772 papers.

Sort:

By area:

  1. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  2. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  3. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,222
  4. 2021ViViT: A Video Vision Transformercited by 1,610
  5. 2021DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharingcited by 1,219
  6. 2022LaMDA: Language Models for Dialog Applicationscited by 1,218
  7. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,209
  8. 2021Unsupervised Dense Information Retrieval with Contrastive Learningcited by 1,203
  9. 2021Perceiver: General Perception with Iterative Attentioncited by 1,089
  10. 2021GLM: General Language Model Pretraining with Autoregressive Blank Infillingcited by 1,026
  11. 2022BERTopic: Neural topic modeling with a class-based TF-IDF procedurecited by 779
  12. 2020Aligning AI With Shared Human Valuescited by 706
38 more
  1. 2023Siren's Song in the AI Ocean: A Survey on Hallucination in Large Language Modelscited by 695
  2. 2023Unifying Large Language Models and Knowledge Graphs: A Roadmapcited by 663
  3. 2023LLM-Pruner: On the Structural Pruning of Large Language Modelscited by 643
  4. 2022ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformerscited by 517
  5. 2024Large Language Models: A Surveycited by 506
  6. 2021Less is More: ClipBERT for Video-and-Language Learning via Sparse Samplingcited by 484
  7. 2022MaxViT: Multi-Axis Vision Transformercited by 476
  8. 2021Nystr"omformer: A Nystr"om-Based Algorithm for Approximating Self-Attentioncited by 468
  9. 2021Recent Advances in Natural Language Processing via Large Pre-Trained Language Models: A Surveycited by 457
  10. 2021Unifying Vision-and-Language Tasks via Text Generationcited by 411
  11. 2021Attention is Not All You Need: Pure Attention Loses Rank Doubly Exponentially with Depthcited by 411
  12. 2021BOLD: Dataset and Metrics for Measuring Biases in Open-Ended Language Generationcited by 410
  13. 2023Harnessing the Power of LLMs in Practice: A Survey on ChatGPT and Beyondcited by 409
  14. 2023GPT-4 Is Too Smart To Be Safe: Stealthy Chat with LLMs via Ciphercited by 401
  15. 2022DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scalecited by 400
  16. 2023Large Language Models for Software Engineering: A Systematic Literature Reviewcited by 389
  17. 2021QA-GNN: Reasoning with Language Models and Knowledge Graphs for Question Answeringcited by 386
  18. 2022BEATs: Audio Pre-Training with Acoustic Tokenizerscited by 378
  19. 2025Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approachcited by 372
  20. 2021Measuring and Improving Consistency in Pretrained Language Modelscited by 361
  21. 2021AdaViT: Adaptive Tokens for Efficient Vision Transformercited by 360
  22. 2021True Few-Shot Learning with Language Modelscited by 343
  23. 2021Choose a Transformer: Fourier or Galerkincited by 333
  24. 2021AgentFormer: Agent-Aware Transformers for Socio-Temporal Multi-Agent Forecastingcited by 327
  25. 2021Do Prompt-Based Models Really Understand the Meaning of their Prompts?cited by 324
  26. 2021Pre-Trained Models: Past, Present and Futurecited by 319
  27. 2021Factual Probing Is [MASK]: Learning vs. Learning to Recallcited by 291
  28. 2022Language Models are Realistic Tabular Data Generatorscited by 290
  29. 2023SqueezeLLM: Dense-and-Sparse Quantizationcited by 285
  30. 2024A Survey on Vision-Language-Action Models for Embodied AIcited by 274
  31. 2021An Empirical Study of Training End-to-End Vision-and-Language Transformerscited by 271
  32. 2024A Comprehensive Survey of Hallucination Mitigation Techniques in Large Language Modelscited by 266
  33. 2021ERNIE 3.0: Large-scale Knowledge Enhanced Pre-training for Language Understanding and Generationcited by 258
  34. 2022Self-Supervised Speech Representation Learning: A Reviewcited by 258
  35. 2023From Pretraining Data to Language Models to Downstream Tasks: Tracking the Trails of Political Biases Leading to Unfair NLP Modelscited by 257
  36. 2021I-BERT: Integer-only BERT Quantizationcited by 250
  37. 2023Auto-GPT for Online Decision Making: Benchmarks and Additional Opinionscited by 250
  38. 2022Selective Annotation Makes Language Models Better Few-Shot Learnerscited by 242

50 most cited of 2,772 All 2,772 (JSON)