mlsanityData to 13 Oct 2026

LLaMA Pro: Progressive LLaMA with Block Expansion

arXiv 2401.02415 ↗First posted 4 Jan 2024cs.CLcited by 100

Not on the map. Why?

Abstract (arXiv)

Humans generally acquire new skills without compromising the old; however, the opposite holds for Large Language Models (LLMs), e.g., from LLaMA to CodeLLaMA. To this end, we propose a new post-pretraining method for LLMs with an expansion of Transformer blocks. We tune the expanded blocks using only new corpus, efficiently and effectively improving the model's knowledge without catastrophic forgetting. In this paper, we experiment on the corpus of code and math, yielding LLaMA Pro-8.3B, a versatile foundation model initialized from LLaMA2-7B, excelling in general tasks, programming, and mathematics. LLaMA Pro and its instruction-following counterpart (LLaMA Pro-Instruct) achieve advanced performance among various benchmarks, demonstrating superiority over existing open models in the LLaMA family and the immense potential of reasoning and addressing diverse tasks as an intelligent agent. Our findings provide valuable insights into integrating natural and programming languages, laying a solid foundation for developing advanced language agents that operate effectively in various environments.

Key facts. Cited by 100 papers in the map (data through 13 Oct 2026). Builds on 55 of its 69 references that are in the map. First posted on arXiv 4 Jan 2024.

Builds on

55 of 69 references.

References · 55

  1. 2020Language Models are Few-Shot Learnerscited by 32,320
  2. 2022Training language models to follow instructions with human feedbackcited by 15,856
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,231
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,979
  5. 2021Training Verifiers to Solve Math Word Problemscited by 9,916
  6. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,502
  7. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,492
  8. 2021Evaluating Large Language Models Trained on Codecited by 8,205
  9. 2020Measuring Massive Multitask Language Understandingcited by 7,876
  10. 2020Scaling Laws for Neural Language Modelscited by 6,494
  11. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,484
  12. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,317
38 more
  1. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,270
  2. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,798
  3. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,213
  4. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,053
  5. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,765
  6. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,716
  7. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,691
  8. 2021Program Synthesis with Large Language Modelscited by 3,573
  9. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,044
  10. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,949
  11. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,379
  12. 2019Fine-Tuning Language Models from Human Preferencescited by 2,139
  13. 2023Code Llama: Open Foundation Models for Codecited by 2,130
  14. 2017Gradient Episodic Memory for Continual Learningcited by 1,882
  15. 2020Don't Stop Pretraining: Adapt Language Models to Domains and Taskscited by 1,556
  16. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,468
  17. 2020ALFWorld: Aligning Text and Embodied Environments for Interactive Learningcited by 1,167
  18. 2023WizardLM: Empowering large pre-trained language models to follow complex instructionscited by 1,120
  19. 2023LIMA: Less Is More for Alignmentcited by 1,091
  20. 2022Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Taskscited by 1,066
  21. 2019BERT Rediscovers the Classical NLP Pipelinecited by 1,049
  22. 2023BloombergGPT: A Large Language Model for Financecited by 932
  23. 2018Riemannian Walk for Incremental Learning: Understanding Forgetting and Intransigencecited by 737
  24. 2023WizardCoder: Empowering Code Large Language Models with Evol-Instructcited by 731
  25. 2023How Far Can Camels Go? Exploring the State of Instruction Tuning on Open Resourcescited by 407
  26. 2023Llemma: An Open Language Model For Mathematicscited by 387
  27. 2021LexGLUE: A Benchmark Dataset for Legal Language Understanding in Englishcited by 303
  28. 2023MINT: Evaluating LLMs in Multi-turn Interaction with Tools and Language Feedbackcited by 294
  29. 2023The Unlocking Spell on Base LLMs: Rethinking Alignment via In-Context Learningcited by 255
  30. 2023How Abilities in Large Language Models are Affected by Supervised Fine-tuning Data Compositioncited by 223
  31. 2023TheoremQA: A Theorem-driven Question Answering datasetcited by 191
  32. 2023Making LLaMA SEE and Draw with SEED Tokenizercited by 184
  33. 2022Fine-tuned Language Models are Continual Learnerscited by 154
  34. 2023Continual Pre-Training of Large Language Models: How to (re)warm your model?cited by 137
  35. 2023LLM360: Towards Fully Transparent Open-Source LLMscited by 102
  36. 2023Learning to Grow Pretrained Models for Efficient Transformer Trainingcited by 81
  37. 2018CLAUDETTE: an Automated Detector of Potentially Unfair Clauses in Online Terms of Servicecited by 72
  38. 2023Lemur: Harmonizing Natural Language and Code for Language Agentscited by 72

50 most cited of 55 All 55 (JSON)

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2024: 38; 2025: 40; 2026: 22.

100 papers.

Sort:

By area:

  1. 2024Large Language Models: A Surveycited by 520
  2. 2024Continual Learning of Large Language Models: A Comprehensive Surveycited by 230
  3. 2024Arcee's MergeKit: A Toolkit for Merging Large Language Modelscited by 192
  4. 2024Simple and Scalable Strategies to Continually Pre-train Large Language Modelscited by 125
  5. 2024Reuse, Don't Retrain: A Recipe for Continued Pretraining of Language Modelscited by 70
  6. 2024Towards Lifelong Learning of Large Language Models: A Surveycited by 62
  7. 2024Upcycling Large Language Models into Mixture of Expertscited by 48
  8. 2024Stacking Your Transformers: A Closer Look at Model Growth for Efficient LLM Pre-Trainingcited by 42
  9. 2024CorDA: Context-Oriented Decomposition Adaptation of Large Language Models for Task-Aware Parameter-Efficient Fine-tuningcited by 42
  10. 2025SEFE: Superficial and Essential Forgetting Eliminator for Multimodal Continual Instruction Tuningcited by 33
  11. 2024Lottery Ticket Adaptation: Mitigating Destructive Interference in LLMscited by 31
  12. 2024CityLLaVA: Efficient Fine-Tuning for VLMs in City Scenariocited by 27
38 more
  1. 2024Understand What LLM Needs: Dual Preference Alignment for Retrieval-Augmented Generationcited by 27
  2. 2024Configurable Foundation Models: Building LLMs from a Modular Perspectivecited by 22
  3. 2024Continual LLaVA: Continual Instruction Tuning in Large Vision-Language Modelscited by 22
  4. 2024Unlocking Memorization in Large Language Models with Dynamic Soft Promptingcited by 21
  5. 2025Transformer Copilot: Learning from The Mistake Log in LLM Fine-tuningcited by 21
  6. 2024Plot2Code: A Comprehensive Benchmark for Evaluating Multi-modal Large Language Models in Code Generation from Scientific Plotscited by 20
  7. 2024HFT: Half Fine-Tuning for Large Language Modelscited by 18
  8. 2024Multilingual Large Language Models: A Systematic Surveycited by 18
  9. 2025Sailor2: Sailing in South-East Asia with Inclusive Multilingual LLMscited by 18
  10. 2024Parameter Efficient Fine-tuning of Self-supervised ViTs without Catastrophic Forgettingcited by 17
  11. 2024MoE-LPR: Multilingual Extension of Large Language Models through Mixture-of-Experts with Language Priors Routingcited by 16
  12. 2025Unlocking the Power of Function Vectors for Characterizing and Mitigating Catastrophic Forgetting in Continual Instruction Tuningcited by 16
  13. 2024FiTv2: Scalable and Improved Flexible Vision Transformer for Diffusion Modelcited by 15
  14. 2025Put the Space of LoRA Initialization to the Extreme to Preserve Pre-trained Knowledgecited by 15
  15. 2025How to Alleviate Catastrophic Forgetting in LLMs Finetuning? Hierarchical Layer-Wise and Element-Wise Regularizationcited by 13
  16. 2025MergeME: Model Merging Techniques for Homogeneous and Heterogeneous MoEscited by 13
  17. 2024Both Matter: Enhancing the Emotional Intelligence of Large Language Models without Compromising the General Intelligencecited by 11
  18. 2025Less, but Better: Efficient Multilingual Expansion for LLMs via Layer-wise Mixture-of-Expertscited by 11
  19. 2025From Seeing to Experiencing: Scaling Navigation Foundation Models with Reinforcement Learningcited by 11
  20. 2024DLO: Dynamic Layer Operation for Efficient Vertical Scaling of LLMscited by 10
  21. 2024CamelEval: Advancing Culturally Aligned Arabic Language Models and Benchmarkscited by 10
  22. 2024Router-Tuning: A Simple and Effective Approach for Enabling Dynamic-Depth in Transformerscited by 10
  23. 2025Llama-3-Nanda-10B-Chat: An Open Generative Large Language Model for Hindicited by 10
  24. 2024Model-GLUE: Democratized LLM Scaling for A Large Model Zoo in the Wildcited by 9
  25. 2024Exploring Forgetting in Large Language Model Pre-Trainingcited by 9
  26. 2025When Continue Learning Meets Multimodal Large Language Model: A Surveycited by 9
  27. 2024Geneverse: A collection of Open-source Multimodal Large Language Models for Genomic and Proteomic Researchcited by 8
  28. 2025Domain Adaptation of Foundation LLMs for e-Commercecited by 8
  29. 2025Speculate, then Collaborate: Fusing Knowledge of Language Models during Decodingcited by 8
  30. 2025Bring Your Own Knowledge: A Survey of Methods for LLM Knowledge Expansioncited by 8
  31. 2024Towards Incremental Learning in Large Language Models: A Critical Reviewcited by 6
  32. 2024Llama SLayer 8B: Shallow Layers Hold the Key to Knowledge Injectioncited by 6
  33. 2025RWKV-X: A Linear Complexity Hybrid Language Modelcited by 6
  34. 2025A Survey of Classification Tasks and Approaches for Legal Contractscited by 6
  35. 2025LESA: Learnable LLM Layer Scaling-Upcited by 5
  36. 2025ADEPT: Continual Pretraining via Adaptive Expansion and Dynamic Decoupled Tuningcited by 5
  37. 2026Beyond Static Models: An Evolving Framework for Continual Learning in Large Language Models across Training Stagescited by 5
  38. 2025SC-LoRA: Balancing Efficient Fine-tuning and Knowledge Preservation via Subspace-Constrained LoRAcited by 4

50 most cited of 100 All 100 (JSON)