mlsanityData to 14 Oct 2026

BLOOM: A 176B-Parameter Open-Access Multilingual Language Model

arXiv 2211.05100 ↗First posted 9 Nov 2022cs.CLcited by 1,984

Not on the map. Why?

Cited by 110 in LLM reasoning, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Large language models (LLMs) have been shown to be able to perform new tasks based on a few demonstrations or natural language instructions. While these capabilities have led to widespread adoption, most LLMs are developed by resource-rich organizations and are frequently kept from the public. As a step towards democratizing this powerful technology, we present BLOOM, a 176B-parameter open-access language model designed and built thanks to a collaboration of hundreds of researchers. BLOOM is a decoder-only Transformer language model that was trained on the ROOTS corpus, a dataset comprising hundreds of sources in 46 natural and 13 programming languages (59 in total). We find that BLOOM achieves competitive performance on a wide variety of benchmarks, with stronger results after undergoing multitask prompted finetuning. To facilitate future research and applications using LLMs, we publicly release our models and code under the Responsible AI License.

Key facts. Cited by 1,984 papers in the map (data through 14 Oct 2026). Builds on 105 of its 171 references that are in the map. First posted on arXiv 9 Nov 2022.

Builds on

105 of 171 references.

References · 105

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  5. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  6. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  7. 2020Scaling Laws for Neural Language Modelscited by 6,586
  8. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,893
  9. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  10. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  11. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,935
  12. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
38 more
  1. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  2. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,509
  3. 2013Distributed Representations of Words and Phrases and their Compositionalitycited by 3,474
  4. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
  5. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,811
  6. 2022Emergent Abilities of Large Language Modelscited by 2,512
  7. 2021Efficiently Modeling Long Sequences with Structured State Spacescited by 2,465
  8. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,411
  9. 2018Deep contextualized word representationscited by 2,290
  10. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,998
  11. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,989
  12. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,868
  13. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,818
  14. 2020GLU Variants Improve Transformercited by 1,804
  15. 2018A Call for Clarity in Reporting BLEU Scorescited by 1,799
  16. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,729
  17. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,470
  18. 2022Competition-Level Code Generation with AlphaCodecited by 1,462
  19. 2019Energy and Policy Considerations for Deep Learning in NLPcited by 1,254
  20. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,202
  21. 2017Mixed Precision Trainingcited by 1,156
  22. 2021Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationcited by 1,103
  23. 2018Model Cards for Model Reportingcited by 1,080
  24. 2022LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scalecited by 906
  25. 2021Probing Classifiers: Promises, Shortcomings, and Advancescited by 876
  26. 2022GLM-130B: An Open Bilingual Pre-trained Modelcited by 843
  27. 2022GPT-NeoX-20B: An Open-Source Autoregressive Language Modelcited by 713
  28. 2020CrowS-Pairs: A Challenge Dataset for Measuring Social Biases in Masked Language Modelscited by 710
  29. 2019Designing and Interpreting Probes with Control Taskscited by 640
  30. 2022MTEB: Massive Text Embedding Benchmarkcited by 640
  31. 2021Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LMcited by 632
  32. 2020HiPPO: Recurrent Memory with Optimal Polynomial Projectionscited by 605
  33. 2017Deep Learning Scaling is Predictable, Empiricallycited by 582
  34. 2021Carbon Emissions and Large Neural Network Trainingcited by 567
  35. 2020Beyond English-Centric Multilingual Machine Translationcited by 561
  36. 2021The FLORES-101 Evaluation Benchmark for Low-Resource and Multilingual Machine Translationcited by 560
  37. 2022Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Modelcited by 559
  38. 2013Generating Sequences With Recurrent Neural Networkscited by 554

50 most cited of 105 All 105 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 35; 2023: 804; 2024: 741; 2025: 285; 2026: 119.

1,984 papers.

Sort:

By area:

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  4. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,418
  5. 2023Qwen Technical Reportcited by 3,344
  6. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,711
  7. 2023A Survey of Large Language Modelscited by 2,576
  8. 2023Code Llama: Open Foundation Models for Codecited by 2,139
  9. 2022Large Language Models Encode Clinical Knowledgecited by 1,830
  10. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understandingcited by 1,471
  11. 2024LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Modelscited by 1,464
  12. 2023The Rise and Potential of Large Language Model Based Agents: A Surveycited by 1,439
38 more
  1. 2023Video-LLaVA: Learning United Visual Representation by Alignment Before Projectioncited by 1,402
  2. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,386
  3. 2023Gorilla: Large Language Model Connected with Massive APIscited by 1,332
  4. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,298
  5. 2023AgentBench: Evaluating LLMs as Agentscited by 1,267
  6. 2023SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shotcited by 1,121
  7. 2024ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Toolscited by 1,121
  8. 2023ImageReward: Learning and Evaluating Human Preferences for Text-to-Image Generationcited by 1,045
  9. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,040
  10. 2024Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Predictioncited by 1,023
  11. 2024DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Modelscited by 867
  12. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Modelscited by 832
  13. 2023RWKV: Reinventing RNNs for the Transformer Eracited by 827
  14. 2023A Simple and Effective Pruning Approach for Large Language Modelscited by 800
  15. 2023StarCoder: may the source be with you!cited by 799
  16. 2023REPLUG: Retrieval-Augmented Black-Box Language Modelscited by 792
  17. 2024Autoregressive Model Beats Diffusion: Llama for Scalable Image Generationcited by 777
  18. 2023The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Onlycited by 763
  19. 2023The Flan Collection: Designing Data and Methods for Effective Instruction Tuningcited by 739
  20. 2023Can Large Language Models Be an Alternative to Human Evaluations?cited by 734
  21. 2023Aligning Large Multimodal Models with Factually Augmented RLHFcited by 712
  22. 2023C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Modelscited by 708
  23. 2023Siren's Song in the AI Ocean: A Survey on Hallucination in Large Language Modelscited by 698
  24. 2023Instruction Tuning with GPT-4cited by 686
  25. 2023RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignmentcited by 686
  26. 2023Baichuan 2: Open Large-scale Language Modelscited by 670
  27. 2023Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Modelscited by 668
  28. 2024A Survey on Large Language Models for Code Generationcited by 658
  29. 2023Safe RLHF: Safe Reinforcement Learning from Human Feedbackcited by 656
  30. 2023LLM-Pruner: On the Structural Pruning of Large Language Modelscited by 650
  31. 2023A Comprehensive Overview of Large Language Modelscited by 648
  32. 2023WizardMath: Empowering Mathematical Reasoning for Large Language Models via Reinforced Evol-Instructcited by 625
  33. 2023MiniGPT-v2: large language model as a unified interface for vision-language multi-task learningcited by 625
  34. 2023Eliciting Latent Predictions from Transformers with the Tuned Lenscited by 623
  35. 2023Instruction Tuning for Large Language Models: A Surveycited by 623
  36. 2024OLMo: Accelerating the Science of Language Modelscited by 613
  37. 2022Towards Reasoning in Large Language Models: A Surveycited by 596
  38. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 575

50 most cited of 1,984 All 1,984 (JSON)