mlsanityData to 7 Oct 2026

LLaMA: Open and Efficient Foundation Language Models

arXiv 2302.13971 ↗First posted 27 Feb 2023cs.CLcited by 13,958

On the map: LLM reasoning → Reasoning benchmarks and evaluation

Abstract (arXiv)

We introduce LLaMA, a collection of foundation language models ranging from 7B to 65B parameters. We train our models on trillions of tokens, and show that it is possible to train state-of-the-art models using publicly available datasets exclusively, without resorting to proprietary and inaccessible datasets. In particular, LLaMA-13B outperforms GPT-3 (175B) on most benchmarks, and LLaMA-65B is competitive with the best models, Chinchilla-70B and PaLM-540B. We release all our models to the research community.

Key facts. Cited by 13,958 papers in the map (data through 7 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 55 of its 80 references that are in the map. First posted on arXiv 27 Feb 2023.

Builds on

55 of 80 references.

References · 55

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2022Training language models to follow instructions with human feedbackcited by 15,712
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  6. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  7. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  8. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  9. 2020Scaling Laws for Neural Language Modelscited by 6,411
  10. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  11. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  12. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
38 more
  1. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  2. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  3. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  4. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  5. 2021Program Synthesis with Large Language Modelscited by 3,509
  6. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  7. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  8. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
  9. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
  10. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,895
  11. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
  12. 2022Emergent Abilities of Large Language Modelscited by 2,483
  13. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  14. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,140
  15. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,114
  16. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,973
  17. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
  18. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,830
  19. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  20. 2020GLU Variants Improve Transformercited by 1,724
  21. 2019Root Mean Square Layer Normalizationcited by 1,425
  22. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,359
  23. 2020RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Modelscited by 1,235
  24. 2022LaMDA: Language Models for Dialog Applicationscited by 1,218
  25. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,199
  26. 2022CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesiscited by 876
  27. 2017RACE: Large-scale ReAding Comprehension Dataset From Examinationscited by 851
  28. 2022GPT-NeoX-20B: An Open-Source Autoregressive Language Modelcited by 711
  29. 2020CrowS-Pairs: A Challenge Dataset for Measuring Social Biases in Masked Language Modelscited by 703
  30. 2022Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Modelcited by 557
  31. 2017Deep Learning Scaling is Predictable, Empiricallycited by 556
  32. 2013Generating Sequences With Recurrent Neural Networkscited by 549
  33. 2019CCNet: Extracting High Quality Monolingual Datasets from Web Crawl Datacited by 544
  34. 2019The Woman Worked as a Babysitter: On Biases in Language Generationcited by 540
  35. 2022InCoder: A Generative Model for Code Infilling and Synthesiscited by 492
  36. 2018Gender Bias in Coreference Resolutioncited by 438
  37. 2013One Billion Word Benchmark for Measuring Progress in Statistical Language Modelingcited by 302
  38. 2022Reducing Activation Recomputation in Large Transformer Modelscited by 300

50 most cited of 55 All 55 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 2733; 2024: 5206; 2025: 4179; 2026: 1840.

13,958 papers.

Sort:

By area:

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  3. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  4. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  5. 2023Visual Instruction Tuningcited by 7,792
  6. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,764
  7. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  8. 2024Qwen2.5 Technical Reportcited by 4,713
  9. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,616
  10. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,151
  11. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,383
  12. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
38 more
  1. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,317
  2. 2023Qwen Technical Reportcited by 3,273
  3. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,155
  4. 2023Mistral 7Bcited by 3,065
  5. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,048
  6. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  7. 2024OpenVLA: An Open-Source Vision-Language-Action Modelcited by 2,777
  8. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  9. 2023A Survey of Large Language Modelscited by 2,537
  10. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,534
  11. 2023A Survey on Large Language Model based Autonomous Agentscited by 2,203
  12. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  13. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,105
  14. 2023Code Llama: Open Foundation Models for Codecited by 2,105
  15. 2023Voyager: An Open-Ended Embodied Agent with Large Language Modelscited by 2,081
  16. 2024Qwen2 Technical Reportcited by 2,076
  17. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,044
  18. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,883
  19. 2023ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIscited by 1,880
  20. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,696
  21. 2025π0.5: a Vision-Language-Action Model with Open-World Generalizationcited by 1,653
  22. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,624
  23. 2024HunyuanVideo: A Systematic Framework For Large Video Generative Modelscited by 1,597
  24. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contextscited by 1,582
  25. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understandingcited by 1,461
  26. 2024LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Modelscited by 1,442
  27. 2023The Rise and Potential of Large Language Model Based Agents: A Surveycited by 1,422
  28. 2023Video-LLaVA: Learning United Visual Representation by Alignment Before Projectioncited by 1,385
  29. 2024Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysiscited by 1,375
  30. 2023CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model Societycited by 1,348
  31. 2024Depth Anything: Unleashing the Power of Large-Scale Unlabeled Datacited by 1,346
  32. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,311
  33. 2023Gorilla: Large Language Model Connected with Massive APIscited by 1,302
  34. 2023LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Daycited by 1,287
  35. 2024Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Dualitycited by 1,283
  36. 2023FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generationcited by 1,259
  37. 2023Inference-Time Intervention: Eliciting Truthful Answers from a Language Modelcited by 1,259
  38. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,234

50 most cited of 13,958 All 13,958 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Amharic LLaMA and LLaVA: Multimodal LLMs for Low Resource Languages
  2. 2025Zebra-Llama: Towards Extremely Efficient Hybrid Models
  3. 2024Enabling High-Sparsity Foundational Llama Models with Efficient Pretraining and Deployment
  4. 2023Accelerating LLaMA Inference by Enabling Intermediate Layer Decoding via Instruction Tuning with LITE
  5. 2025TFD: A Comprehensive Structured Tibetan Foundation Dataset for Low-Resource Language Processing and Large-Scale Modeling
  6. 2025Llamba: Scaling Distilled Recurrent Models for Efficient Language Processing
  7. 2025Leveraging the true depth of LLMs
  8. 2024Model-GLUE: Democratized LLM Scaling for A Large Model Zoo in the Wild