mlsanityData to 14 Oct 2026

Code Llama: Open Foundation Models for Code

arXiv 2308.12950 ↗First posted 24 Aug 2023cs.CLcited by 2,139

On the map: LLM reasoning → Reasoning benchmarks and evaluation

Abstract (arXiv)

We release Code Llama, a family of large language models for code based on Llama 2 providing state-of-the-art performance among open models, infilling capabilities, support for large input contexts, and zero-shot instruction following ability for programming tasks. We provide multiple flavors to cover a wide range of applications: foundation models (Code Llama), Python specializations (Code Llama - Python), and instruction-following models (Code Llama - Instruct) with 7B, 13B, 34B and 70B parameters each. All models are trained on sequences of 16k tokens and show improvements on inputs with up to 100k tokens. 7B, 13B and 70B Code Llama and Code Llama - Instruct variants support infilling based on surrounding content. Code Llama reaches state-of-the-art performance among open models on several code benchmarks, with scores of up to 67% and 65% on HumanEval and MBPP, respectively. Notably, Code Llama - Python 7B outperforms Llama 2 70B on HumanEval and MBPP, and all our models outperform every other publicly available model on MultiPL-E. We release Code Llama under a permissive license that allows for both research and commercial use.

Key facts. Cited by 2,139 papers in the map (data through 14 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 72 of its 92 references that are in the map. First posted on arXiv 24 Aug 2023.

Builds on

72 of 92 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023LLaMA: Open and Efficient Foundation Language Models27 Feb

Other references · 71

  1. 2017Attention Is All You Needcited by 60,361
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2022Training language models to follow instructions with human feedbackcited by 15,971
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  5. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  6. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  7. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  8. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  9. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  10. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  11. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
  12. 2021Program Synthesis with Large Language Modelscited by 3,637
38 more
  1. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  2. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,455
  3. 2020Longformer: The Long-Document Transformercited by 3,413
  4. 2022Training Compute-Optimal Large Language Modelscited by 3,019
  5. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,975
  6. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,918
  7. 2019The Curious Case of Neural Text Degenerationcited by 2,916
  8. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,989
  9. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,984
  10. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,850
  11. 2022Competition-Level Code Generation with AlphaCodecited by 1,462
  12. 2020CodeBERT: A Pre-Trained Model for Programming and Natural Languagescited by 1,310
  13. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,202
  14. 2021Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationcited by 1,103
  15. 2018Model Cards for Model Reportingcited by 1,080
  16. 2021CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generationcited by 906
  17. 2021Measuring Coding Challenge Competence With APPScited by 902
  18. 2022CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesiscited by 885
  19. 2023StarCoder: may the source be with you!cited by 799
  20. 2023The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Onlycited by 763
  21. 2021CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generationcited by 744
  22. 2023Extending Context Window of Large Language Models via Positional Interpolationcited by 740
  23. 2022GPT-NeoX-20B: An Open-Source Autoregressive Language Modelcited by 713
  24. 2020GraphCodeBERT: Pre-training Code Representations with Data Flowcited by 609
  25. 2023Textbooks Are All You Needcited by 569
  26. 2022InCoder: A Generative Model for Code Infilling and Synthesiscited by 498
  27. 2022ToxiGen: A Large-Scale Machine-Generated Dataset for Adversarial and Implicit Hate Speech Detectioncited by 422
  28. 2021BOLD: Dataset and Metrics for Measuring Biases in Open-Ended Language Generationcited by 412
  29. 2022Unnatural Instructions: Tuning Language Models with (Almost) No Human Laborcited by 397
  30. 2022CodeT: Code Generation with Generated Testscited by 384
  31. 2021Unified Pre-training for Program Understanding and Generationcited by 366
  32. 2022CodeRL: Mastering Code Generation through Pretrained Models and Deep Reinforcement Learningcited by 358
  33. 2023The Impact of Positional Encoding on Length Generalization in Transformerscited by 327
  34. 2020Unsupervised Translation of Programming Languagescited by 263
  35. 2022Efficient Training of Language Models to Fill in the Middlecited by 255
  36. 2023LongNet: Scaling Transformers to 1,000,000,000 Tokenscited by 204
  37. 2023Planning with Large Language Models for Code Generationcited by 182
  38. 2023SantaCoder: don't reach for the stars!cited by 170

50 most cited of 71 All 71 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 183; 2024: 952; 2025: 651; 2026: 353.

2,139 papers.

Sort:

By area:

  1. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,373
  2. 2023Qwen Technical Reportcited by 3,344
  3. 2023Mistral 7Bcited by 3,164
  4. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
  5. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 2,004
  6. 2023Jailbreaking Black Box Large Language Models in Twenty Queriescited by 1,420
  7. 2024Qwen2.5-Coder Technical Reportcited by 1,203
  8. 2024DeepSeek-Coder: When the Large Language Model Meets Programming -- The Rise of Code Intelligencecited by 1,181
  9. 2023Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!cited by 1,154
  10. 2024Chameleon: Mixed-Modal Early-Fusion Foundation Modelscited by 927
  11. 2023MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Modelscited by 735
  12. 2023Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunchcited by 694
38 more
  1. 2024A Survey on Large Language Models for Code Generationcited by 658
  2. 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Servingcited by 617
  3. 2025Why Do Multi-Agent LLM Systems Fail?cited by 604
  4. 2023YaRN: Efficient Context Window Extension of Large Language Modelscited by 601
  5. 2023Eureka: Human-Level Reward Design via Coding Large Language Modelscited by 571
  6. 2024Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Modelscited by 557
  7. 2023Identifying the Risks of LM Agents with an LM-Emulated Sandboxcited by 548
  8. 2024BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructionscited by 546
  9. 2023The Falcon Series of Open Language Modelscited by 535
  10. 2024Measuring Multimodal Mathematical Reasoning with MATH-Vision Datasetcited by 526
  11. 2024Large Language Models: A Surveycited by 525
  12. 2024FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precisioncited by 519
  13. 2023MAmmoTH: Building Math Generalist Models through Hybrid Instruction Tuningcited by 487
  14. 2024SpinQuant: LLM quantization with learned rotationscited by 471
  15. 2023LMSYS-Chat-1M: A Large-Scale Real-World LLM Conversation Datasetcited by 461
  16. 2024Mixture-of-Agents Enhances Large Language Model Capabilitiescited by 438
  17. 2024PyramidKV: Dynamic KV Cache Compression based on Pyramidal Information Funnelingcited by 432
  18. 2023Cognitive Architectures for Language Agentscited by 429
  19. 2024Agentless: Demystifying LLM-based Software Engineering Agentscited by 421
  20. 2023Llemma: An Open Language Model For Mathematicscited by 388
  21. 2023Text-to-SQL Empowered by Large Language Models: A Benchmark Evaluationcited by 385
  22. 2025Towards Reasoning Era: A Survey of Long Chain-of-Thought for Reasoning Large Language Modelscited by 384
  23. 2024MLLM-as-a-Judge: Assessing Multimodal LLM-as-a-Judge with Vision-Language Benchmarkcited by 372
  24. 2024Retrieval-Augmented Generation for AI-Generated Content: A Surveycited by 354
  25. 2024AgentBoard: An Analytical Evaluation Board of Multi-turn LLM Agentscited by 339
  26. 2024DeepSeek-Coder-V2: Breaking the Barrier of Closed-Source Models in Code Intelligencecited by 331
  27. 2024LoRA Learns Less and Forgets Lesscited by 328
  28. 2024Discrete Flow Matchingcited by 317
  29. 2023Effective Long-Context Scaling of Foundation Modelscited by 303
  30. 2023Improving Text Embeddings with Large Language Modelscited by 303
  31. 2024CRUXEval: A Benchmark for Code Reasoning, Understanding and Executioncited by 300
  32. 2024Break the Sequential Dependency of LLM Inference Using Lookahead Decodingcited by 298
  33. 2023MINT: Evaluating LLMs in Multi-turn Interaction with Tools and Language Feedbackcited by 297
  34. 2024Large Language Models for Mathematical Reasoning: Progresses and Challengescited by 284
  35. 2024A Survey on Knowledge Distillation of Large Language Modelscited by 284
  36. 2023ToRA: A Tool-Integrated Reasoning Agent for Mathematical Problem Solvingcited by 282
  37. 2024CodeAgent: Enhancing Code Generation with Tool-Integrated Agent Systems for Real-World Repo-level Coding Challengescited by 282
  38. 2024Scaling Diffusion Language Models via Adaptation from Autoregressive Modelscited by 274

50 most cited of 2,139 All 2,139 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Which Quantization Should I Use? A Unified Evaluation of llama.cpp Quantization on Llama-3.1-8B-Instruct
  2. 2025Evolution of meta's llama models and parameter-efficient fine-tuning of large language models: a survey
  3. 2023Gorilla: Large Language Model Connected with Massive APIs
  4. 2025Llama-based source code vulnerability detection: Prompt engineering vs Fine tuning
  5. 2026Benchmarking LLAMA Model Security Against OWASP Top 10 For LLM Applications
  6. 2023Accelerating LLaMA Inference by Enabling Intermediate Layer Decoding via Instruction Tuning with LITE
  7. 2024StructuredRAG: JSON Response Formatting with Large Language Models
  8. 2024The Uniqueness of LLaMA3-70B Series with Per-Channel Quantization