mlsanityData to 7 Oct 2026

Efficient Continual Pre-training for Building Domain Specific Large Language Models

arXiv 2311.08545 ↗First posted 14 Nov 2023cs.CLcited by 33

Not on the map. Why?

Abstract (arXiv)

Large language models (LLMs) have demonstrated remarkable open-domain capabilities. LLMs tailored for a domain are typically trained entirely on domain corpus to excel at handling domain-specific tasks. In this work, we explore an alternative strategy of continual pre-training as a means to develop domain-specific LLMs over an existing open-domain LLM. We introduce FinPythia-6.9B, developed through domain-adaptive continual pre-training on the financial domain. Continual pre-trained FinPythia showcases consistent improvements on financial tasks over the original foundational model. We further explore simple but effective data selection strategies for continual pre-training. Our data selection strategies outperform vanilla continual pre-training's performance with just 10% of corpus size and cost, without any degradation on open-domain standard tasks. Our work proposes an alternative solution to building domain-specific LLMs cost-effectively.

Key facts. Cited by 33 papers in the map (data through 7 Oct 2026). Builds on 32 of its 48 references that are in the map. First posted on arXiv 14 Nov 2023.

Builds on

32 of 48 references.

References · 32

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  3. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  4. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  5. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  6. 2015Domain-Adversarial Training of Neural Networkscited by 3,023
  7. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
  8. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
  9. 2022Large Language Models Encode Clinical Knowledgecited by 1,807
  10. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,734
  11. 2020Don't Stop Pretraining: Adapt Language Models to Domains and Taskscited by 1,548
  12. 2023BloombergGPT: A Large Language Model for Financecited by 923
20 more
  1. 2022Galactica: A Large Language Model for Sciencecited by 736
  2. 2021Deduplicating Training Data Makes Language Models Bettercited by 705
  3. 2023Towards Expert-Level Medical Question Answering with Large Language Modelscited by 382
  4. 2023Data Selection for Language Models via Importance Resamplingcited by 322
  5. 2013Good Debt or Bad Debt: Detecting Semantic Orientations in Economic Textscited by 310
  6. 2023PIXIU: A Large Language Model, Instruction Data and Evaluation Benchmark for Financecited by 217
  7. 2023Continual Pre-training of Language Modelscited by 194
  8. 2021Towards Continual Knowledge Learning of Language Modelscited by 172
  9. 2020Unsupervised Domain Clusters in Pretrained Language Modelscited by 160
  10. 2022TemporalWiki: A Lifelong Benchmark for Training and Evaluating Ever-Evolving Language Modelscited by 117
  11. 2021Lifelong Pretraining: Continually Adapting Language Models to Emerging Corporacited by 114
  12. 2022Same Pre-training Loss, Better Downstream: Implicit Bias Matters for Language Modelscited by 84
  13. 2020Impact of News on the Commodity Market: Dataset and Resultscited by 80
  14. 2017Learning to select data for transfer learning with Bayesian Optimizationcited by 64
  15. 2018A Systematic Classification of Knowledge, Reasoning, and Context within the ARC Datasetcited by 54
  16. 2021EDGAR-CORPUS: Billions of Tokens Make The World Go Roundcited by 32
  17. 2023Adapting a Language Model While Preserving its General Knowledgecited by 23
  18. 2016Learning the Curriculum with Bayesian Optimization for Task-Specific Word Representation Learningcited by 22
  19. 2019Using Similarity Measures to Select Pretraining Data for NERcited by 15
  20. 2022A Word is Worth A Thousand Dollars: Adversarial Attack on Tweets Fools Stock Predictionscited by 15

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 18; 2025: 8; 2026: 7.

33 papers.

Sort:

By area:

  1. 2024Continual Learning of Large Language Models: A Comprehensive Surveycited by 228
  2. 2024Continual Learning for Large Language Models: A Surveycited by 144
  3. 2024Simple and Scalable Strategies to Continually Pre-train Large Language Modelscited by 122
  4. 2024From Matching to Generation: A Survey on Generative Information Retrievalcited by 89
  5. 2025Nemotron-CLIMB: CLustering-based Iterative Data Mixture Bootstrapping for Language Model Pre-trainingcited by 62
  6. 2024Recent Advances of Foundation Language Models-based Continual Learning: A Surveycited by 46
  7. 2024Unveiling the Secret Recipe: A Guide For Supervised Fine-Tuning Small LLMscited by 39
  8. 2024Tele-LLMs: A Series of Specialized Large Language Models for Telecommunicationscited by 25
  9. 2024Continual LLaVA: Continual Instruction Tuning in Large Vision-Language Modelscited by 22
  10. 2024Self-Updatable Large Language Models by Integrating Context into Model Parameterscited by 21
  11. 2025The Future of Continual Learning in the Era of Foundation Models: Three Key Directionscited by 16
  12. 2024Mixing It Up: The Cocktail Effect of Multi-Task Fine-Tuning on LLM Performance -- A Case Study in Financecited by 15
21 more
  1. 2024Towards LifeSpan Cognitive Systemscited by 13
  2. 2024Extend Model Merging from Fine-Tuned to Pre-Trained Large Language Models via Weight Disentanglementcited by 11
  3. 2024Delving into the Reversal Curse: How Far Can Large Language Models Generalize?cited by 11
  4. 2026Data-Efficient Autoregressive-to-Diffusion Language Models via On-Policy Distillationcited by 11
  5. 2024Task Oriented In-Domain Data Augmentationcited by 6
  6. 2026MDAgent2: Large Language Model for Code Generation and Knowledge Q&A in Molecular Dynamicscited by 6
  7. 2024Foundation Models for Low-Resource Language Education (Vision Paper)cited by 5
  8. 2025Requirements Elicitation Follow-Up Question Generationcited by 5
  9. 2025Decoupling Knowledge and Reasoning in LLMs: An Exploration Using Cognitive Dual-System Theorycited by 4
  10. 2026Beyond Static Models: An Evolving Framework for Continual Learning in Large Language Models across Training Stagescited by 4
  11. 2024Developing Cryptocurrency Trading Strategy Based on Autoencoder-CNN-GANs Algorithmscited by 3
  12. 2025Towards EnergyGPT: A Large Language Model Specialized for the Energy Sectorcited by 3
  13. 2024TapWeight: Reweighting Pretraining Objectives for Task-Adaptive Pretrainingcited by 1
  14. 2026Diffract: Spectral View of LLM Domain Adaptationcited by 1
  15. 2024ZIP-FIT: Embedding-Free Data Selection via Compression-Based Alignmentcited by 0
  16. 2025Continually Evolved Multimodal Foundation Models for Cancer Prognosiscited by 0
  17. 2025AI-Driven Generation of Old English: A Framework for Low-Resource Languagescited by 0
  18. 2025Evontree: Ontology Rule-Guided Self-Evolution of Large Language Modelscited by 0
  19. 2026Cognitively Layered Data Synthesis for Domain Adaptation of LLMs to Space Situational Awarenesscited by 0
  20. 2026LegalMidm: Use-Case-Driven Legal Domain Specialization for Korean Large Language Modelcited by 0
  21. 2026Mix-MoE: Improving Multilingual Machine Translation of Large Language Models through Mixed MoEscited by 0