mlsanityData to 7 Oct 2026

The Falcon Series of Open Language Models

arXiv 2311.16867 ↗First posted 28 Nov 2023cs.CL cs.AIcited by 526

Not on the map. Why?

Cited by 42 in LLM reasoning, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We introduce the Falcon series: 7B, 40B, and 180B parameters causal decoder-only models trained on a diverse high-quality corpora predominantly assembled from web data. The largest model, Falcon-180B, has been trained on over 3.5 trillion tokens of text--the largest openly documented pretraining run. Falcon-180B significantly outperforms models such as PaLM or Chinchilla, and improves upon concurrently developed models such as LLaMA 2 or Inflection-1. It nears the performance of PaLM-2-Large at a reduced pretraining and inference cost, making it, to our knowledge, one of the three best language models in the world along with GPT-4 and PaLM-2-Large. We report detailed evaluations, as well as a deep dive into the methods and custom tooling employed to pretrain Falcon. Notably, we report on our custom distributed training codebase, allowing us to efficiently pretrain these models on up to 4,096 A100s on cloud AWS infrastructure with limited interconnect. We release a 600B tokens extract of our web dataset, as well as the Falcon-7/40/180B models under a permissive license to foster open-science and accelerate the development of an open ecosystem of large language models.

Key facts. Cited by 526 papers in the map (data through 7 Oct 2026). Builds on 105 of its 173 references that are in the map. First posted on arXiv 28 Nov 2023.

Builds on

105 of 173 references.

References · 105

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2023GPT-4 Technical Reportcited by 18,132
  5. 2022Training language models to follow instructions with human feedbackcited by 15,712
  6. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  7. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  8. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  9. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  10. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  11. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  12. 2021Evaluating Large Language Models Trained on Codecited by 8,053
38 more
  1. 2020Scaling Laws for Neural Language Modelscited by 6,411
  2. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  3. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  4. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  5. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,530
  6. 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,346
  7. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  8. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  9. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  10. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  11. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  12. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  13. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
  14. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
  15. 2022Emergent Abilities of Large Language Modelscited by 2,483
  16. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  17. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,140
  18. 2023Code Llama: Open Foundation Models for Codecited by 2,105
  19. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,970
  20. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
  21. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  22. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,734
  23. 2020GLU Variants Improve Transformercited by 1,724
  24. 2017Decoupled Weight Decay Regularizationcited by 1,722
  25. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,718
  26. 2016Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translationcited by 1,471
  27. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,462
  28. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,359
  29. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,311
  30. 2022No Language Left Behind: Scaling Human-Centered Machine Translationcited by 1,290
  31. 2022LaMDA: Language Models for Dialog Applicationscited by 1,218
  32. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,199
  33. 2023PaLM 2 Technical Reportcited by 1,154
  34. 2018Self-Attention with Relative Position Representationscited by 1,112
  35. 2021Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationcited by 1,085
  36. 2023Large Language Models are not Fair Evaluatorscited by 1,050
  37. 2023StarCoder: may the source be with you!cited by 790
  38. 2022Quantifying Memorization Across Neural Language Modelscited by 755

50 most cited of 105 All 105 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 4; 2024: 285; 2025: 164; 2026: 73.

526 papers.

Sort:

By area:

  1. 2024Qwen2.5 Technical Reportcited by 4,713
  2. 2024LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Modelscited by 1,442
  3. 2024MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategiescited by 694
  4. 2024OLMo: Accelerating the Science of Language Modelscited by 599
  5. 2024Moshi: a speech-text foundation model for real-time dialoguecited by 536
  6. 2024Measuring Multimodal Mathematical Reasoning with MATH-Vision Datasetcited by 516
  7. 2024BioMistral: A Collection of Open-Source Pretrained Large Language Models for Medical Domainscited by 299
  8. 2024OLMoE: Open Mixture-of-Experts Language Modelscited by 291
  9. 2024Massive Activations in Large Language Modelscited by 277
  10. 2024MobileLLM: Optimizing Sub-billion Parameter Language Models for On-Device Use Casescited by 216
  11. 2024A Survey on Efficient Inference for Large Language Modelscited by 197
  12. 2024RedPajama: an Open Dataset for Training Large Language Modelscited by 178
38 more
  1. 2024RAID: A Shared Benchmark for Robust Evaluation of Machine-Generated Text Detectorscited by 157
  2. 2024Kernel Language Entropy: Fine-grained Uncertainty Quantification for LLMs from Semantic Similaritiescited by 156
  3. 2024FinBen: A Holistic Financial Benchmark for Large Language Modelscited by 140
  4. 2024The Well: a Large-Scale Collection of Diverse Physics Simulations for Machine Learningcited by 134
  5. 2025LLM Post-Training: A Deep Dive into Reasoning Large Language Modelscited by 129
  6. 2024Observational Scaling Laws and the Predictability of Language Model Performancecited by 128
  7. 2024Simple and Scalable Strategies to Continually Pre-train Large Language Modelscited by 122
  8. 2024Scaling Laws with Vocabulary: Larger Models Deserve Larger Vocabulariescited by 113
  9. 2024Benchmarking Large Language Models on Answering and Explaining Challenging Medical Questionscited by 106
  10. 2024On-Device Language Models: A Comprehensive Reviewcited by 101
  11. 2024OLMES: A Standard for Language Model Evaluationscited by 100
  12. 2024ALERT: A Comprehensive Benchmark for Assessing Large Language Models' Safety through Red Teamingcited by 96
  13. 2024CRAG -- Comprehensive RAG Benchmarkcited by 94
  14. 2024Attacking Large Language Models with Projected Gradient Descentcited by 91
  15. 2024Large Concept Models: Language Modeling in a Sentence Representation Spacecited by 86
  16. 2024Fast Adversarial Attacks on Language Models In One GPU Minutecited by 85
  17. 2024A Systematic Survey and Critical Review on Evaluating Large Language Models: Challenges, Limitations, and Recommendationscited by 84
  18. 2024On Verbalized Confidence Scores for LLMscited by 83
  19. 2024When LLMs Meet Cybersecurity: A Systematic Literature Reviewcited by 70
  20. 2024Large language models surpass human experts in predicting neuroscience resultscited by 69
  21. 2024ReLU2 Wins: Discovering Efficient Activation Functions for Sparse LLMscited by 66
  22. 2024Instruction Pre-Training: Language Models are Supervised Multitask Learnerscited by 63
  23. 2024Empowering LLMs to Understand and Generate Complex Vector Graphicscited by 63
  24. 2024Aligning Large Language Models with Representation Editing: A Control Perspectivecited by 62
  25. 2024Towards Cross-Tokenizer Distillation: the Universal Logit Distillation Loss for LLMscited by 61
  26. 2024Semantics-Adaptive Activation Intervention for LLMs via Dynamic Steering Vectorscited by 61
  27. 2024Factual Confidence of LLMs: on Reliability and Robustness of Current Estimatorscited by 59
  28. 2024Compression Represents Intelligence Linearlycited by 56
  29. 2024Safetywashing: Do AI Safety Benchmarks Actually Measure Safety Progress?cited by 56
  30. 2024POSIX: A Prompt Sensitivity Index For Large Language Modelscited by 56
  31. 2025DeepScientist: Advancing Frontier-Pushing Scientific Findings Progressivelycited by 56
  32. 2024From Words to Numbers: Your Large Language Model Is Secretly A Capable Regressor When Given In-Context Examplescited by 55
  33. 2024Elements of World Knowledge (EWoK): A Cognition-Inspired Framework for Evaluating Basic World Knowledge in Language Modelscited by 55
  34. 2024LLM-Seg: Bridging Image Segmentation and Large Language Model Reasoningcited by 52
  35. 2024Understanding the Capabilities and Limitations of Large Language Models for Cultural Commonsensecited by 50
  36. 2024PRP: Propagating Universal Perturbations to Attack Large Language Model Guard-Railscited by 49
  37. 2024Dated Data: Tracing Knowledge Cutoffs in Large Language Modelscited by 48
  38. 2024Disclosure and Mitigation of Gender Bias in LLMscited by 47

50 most cited of 526 All 526 (JSON)