mlsanityData to 7 Oct 2026

Galactica: A Large Language Model for Science

arXiv 2211.09085 ↗First posted 16 Nov 2022cs.CL stat.MLcited by 736

Not on the map. Why?

Cited by 94 in LLM reasoning, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Information overload is a major obstacle to scientific progress. The explosive growth in scientific literature and data has made it ever harder to discover useful insights in a large mass of information. Today scientific knowledge is accessed through search engines, but they are unable to organize scientific knowledge alone. In this paper we introduce Galactica: a large language model that can store, combine and reason about scientific knowledge. We train on a large scientific corpus of papers, reference material, knowledge bases and many other sources. We outperform existing models on a range of scientific tasks. On technical knowledge probes such as LaTeX equations, Galactica outperforms the latest GPT-3 by 68.2% versus 49.0%. Galactica also performs well on reasoning, outperforming Chinchilla on mathematical MMLU by 41.3% to 35.7%, and PaLM 540B on MATH with a score of 20.4% versus 8.8%. It also sets a new state-of-the-art on downstream tasks such as PubMedQA and MedMCQA dev of 77.6% and 52.9%. And despite not being trained on a general corpus, Galactica outperforms BLOOM and OPT-175B on BIG-bench. We believe these results demonstrate the potential for language models as a new interface for science. We open source the model for the benefit of the scientific community.

Key facts. Cited by 736 papers in the map (data through 7 Oct 2026). Builds on 68 of its 107 references that are in the map. First posted on arXiv 16 Nov 2022.

Builds on

68 of 107 references.

References · 68

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  6. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  7. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  8. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  9. 2020Scaling Laws for Neural Language Modelscited by 6,411
  10. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  11. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  12. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
38 more
  1. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  2. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  3. 2016Gaussian Error Linear Units (GELUs)cited by 3,431
  4. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  5. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  6. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
  7. 2017Billion-scale similarity search with GPUscited by 2,477
  8. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,140
  9. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,114
  10. 2019Language Models as Knowledge Bases?cited by 1,922
  11. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,793
  12. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,767
  13. 2017Decoupled Weight Decay Regularizationcited by 1,722
  14. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,462
  15. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,359
  16. 2021Improving language models by retrieving from trillions of tokenscited by 1,278
  17. 2019PubMedQA: A Dataset for Biomedical Research Question Answeringcited by 1,271
  18. 2020RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Modelscited by 1,235
  19. 2022LaMDA: Language Models for Dialog Applicationscited by 1,218
  20. 2021Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationcited by 1,085
  21. 2020Language (Technology) is Power: A Critical Survey of "Bias" in NLPcited by 951
  22. 2022Atlas: Few-shot Learning with Retrieval Augmented Language Modelscited by 940
  23. 2020StereoSet: Measuring stereotypical bias in pretrained language modelscited by 911
  24. 2016Bag of Tricks for Efficient Text Classificationcited by 869
  25. 2022GPT-NeoX-20B: An Open-Source Autoregressive Language Modelcited by 711
  26. 2021Deduplicating Training Data Makes Language Models Bettercited by 705
  27. 2020CrowS-Pairs: A Challenge Dataset for Measuring Social Biases in Masked Language Modelscited by 703
  28. 2019The Woman Worked as a Babysitter: On Biases in Language Generationcited by 540
  29. 2018Wizard of Wikipedia: Knowledge-Powered Conversational agentscited by 530
  30. 2022MedMCQA : A Large-scale Multi-Subject Multi-Choice Dataset for Medical domain Question Answeringcited by 525
  31. 2016Adaptive Computation Time for Recurrent Neural Networkscited by 479
  32. 2020UnifiedQA: Crossing Format Boundaries With a Single QA Systemcited by 475
  33. 2021A Dataset of Information-Seeking Questions and Answers Anchored in Research Paperscited by 475
  34. 2022Memory-Based Model Editing at Scalecited by 443
  35. 2019Measuring Bias in Contextualized Word Representationscited by 275
  36. 2019QASC: A Dataset for Question Answering via Sentence Compositioncited by 248
  37. 2022LinkBERT: Pretraining Language Models with Document Linkscited by 209
  38. 2021Societal Biases in Language Generation: Progress and Challengescited by 201

50 most cited of 68 All 68 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 8; 2023: 237; 2024: 249; 2025: 142; 2026: 100.

736 papers.

Sort:

By area:

  1. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  2. 2023A Survey of Large Language Modelscited by 2,537
  3. 2022Large Language Models Encode Clinical Knowledgecited by 1,807
  4. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contextscited by 1,582
  5. 2023A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questionscited by 1,582
  6. 2024The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scalecited by 1,104
  7. 2023BloombergGPT: A Large Language Model for Financecited by 923
  8. 2023CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquingcited by 751
  9. 2023MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Modelscited by 724
  10. 2023C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Modelscited by 694
  11. 2023Baichuan 2: Open Large-scale Language Modelscited by 664
  12. 2023A Comprehensive Overview of Large Language Modelscited by 639
38 more
  1. 2023WizardMath: Empowering Mathematical Reasoning for Large Language Models via Reinforced Evol-Instructcited by 618
  2. 2024KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cachecited by 604
  3. 2022Towards Reasoning in Large Language Models: A Surveycited by 593
  4. 2024Large Language Models: A Surveycited by 506
  5. 2023Faith and Fate: Limits of Transformers on Compositionalitycited by 496
  6. 2023MAmmoTH: Building Math Generalist Models through Hybrid Instruction Tuningcited by 483
  7. 2023Augmented Language Models: a Surveycited by 398
  8. 2023Trustworthy LLMs: a Survey and Guideline for Evaluating Large Language Models' Alignmentcited by 394
  9. 2023Scaling Data-Constrained Language Modelscited by 391
  10. 2023Large Language Models for Software Engineering: A Systematic Literature Reviewcited by 389
  11. 2023Llemma: An Open Language Model For Mathematicscited by 385
  12. 2024ReST-MCTS*: LLM Self-Training via Process Reward Guided Tree Searchcited by 338
  13. 2024Benchmarking Retrieval-Augmented Generation for Medicinecited by 326
  14. 2023The Impact of Positional Encoding on Length Generalization in Transformerscited by 319
  15. 2023Alphazero-like Tree-Search can Guide Large Language Model Decoding and Trainingcited by 291
  16. 2024Large Language Models for Mathematical Reasoning: Progresses and Challengescited by 283
  17. 2024A Survey on Knowledge Distillation of Large Language Modelscited by 276
  18. 2023MEDITRON-70B: Scaling Medical Pretraining for Large Language Modelscited by 272
  19. 2023Challenges and Applications of Large Language Modelscited by 251
  20. 2023FacTool: Factuality Detection in Generative AI -- A Tool Augmented Framework for Multi-Task and Multi-Domain Scenarioscited by 247
  21. 2023SciBench: Evaluating College-Level Scientific Problem-Solving Abilities of Large Language Modelscited by 246
  22. 2023VerilogEval: Evaluating Large Language Models for Verilog Code Generationcited by 246
  23. 2024OpenCodeInterpreter: Integrating Code Generation with Execution and Refinementcited by 242
  24. 2023Nougat: Neural Optical Understanding for Academic Documentscited by 240
  25. 2023One for All: Towards Training One Graph Model for All Classification Taskscited by 239
  26. 2024ReFT: Reasoning with Reinforced Fine-Tuningcited by 236
  27. 2024Continual Learning of Large Language Models: A Comprehensive Surveycited by 228
  28. 2024Mastering Text-to-Image Diffusion: Recaptioning, Planning, and Generating with Multimodal LLMscited by 224
  29. 2023A Comprehensive Survey of AI-Generated Content (AIGC): A History of Generative AI from GAN to ChatGPTcited by 221
  30. 2024MobileLLM: Optimizing Sub-billion Parameter Language Models for On-Device Use Casescited by 216
  31. 2022Is GPT-3 a Good Data Annotator?cited by 208
  32. 2024Tower: An Open Multilingual Large Language Model for Translation-Related Taskscited by 205
  33. 2025ZeroSearch: Incentivize the Search Capability of LLMs without Searchingcited by 203
  34. 2023Large Language Models on Graphs: A Comprehensive Surveycited by 190
  35. 2023TheoremQA: A Theorem-driven Question Answering datasetcited by 189
  36. 2023TaskMatrix.AI: Completing Tasks by Connecting Foundation Models with Millions of APIscited by 184
  37. 2023Mitigating the Alignment Tax of RLHFcited by 181
  38. 2023What can Large Language Models do in chemistry? A comprehensive benchmark on eight taskscited by 179

50 most cited of 736 All 736 (JSON)