mlsanityData to 14 Oct 2026

Efficient Domain-adaptive Continual Pretraining for the Process Industry in the German Language

arXiv 2504.19856 ↗First posted 28 Apr 2025cs.CLcited by 3

Not on the map. Why?

Abstract (arXiv)

Domain-adaptive continual pretraining (DAPT) is a state-of-the-art technique that further trains a language model (LM) on its pretraining task, e.g., masked language modeling (MLM), when common domain adaptation via LM fine-tuning is not possible due to a lack of labeled task data. Although popular, MLM requires a significant corpus of domain-related data, which is difficult to obtain for specific domains in languages other than English, such as the process industry in the German language. This paper introduces an efficient approach called ICL-augmented pretraining or ICL-APT that leverages in-context learning (ICL) and k-nearest neighbors (kNN) to augment target data with domain-related and in-domain texts, significantly reducing GPU time while maintaining strong model performance. Our results show that the best configuration of ICL-APT performed better than the state-of-the-art DAPT by 28.7% (7.87 points) and requires almost 4 times less GPU-computing time, providing a cost-effective solution for industries with limited computational capacity. The findings highlight the broader applicability of this framework to other low-resource industries, making NLP-based solutions more accessible and feasible in production environments.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 21 of its 37 references that are in the map. First posted on arXiv 28 Apr 2025.

Builds on

21 of 37 references.

References · 21

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2016MS MARCO: A Human Generated MAchine Reading COmprehension Datasetcited by 1,605
  3. 2020Don't Stop Pretraining: Adapt Language Models to Domains and Taskscited by 1,565
  4. 2019SciBERT: A Pretrained Language Model for Scientific Textcited by 1,414
  5. 2021BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Modelscited by 1,252
  6. 2020The State and Fate of Linguistic Diversity and Inclusion in the NLP Worldcited by 798
  7. 2020Making Monolingual Sentence Embeddings Multilingual using Knowledge Distillationcited by 593
  8. 2023Data Selection for Language Models via Importance Resamplingcited by 329
  9. 2023Continual Pre-training of Language Modelscited by 198
  10. 2020A Survey on Recent Approaches for Natural Language Processing in Low-Resource Scenarioscited by 171
  11. 2020German's Next Language Modelcited by 144
  12. 2023Few-shot Fine-tuning vs. In-context Learning: A Fair Comparison and Evaluationcited by 136
9 more
  1. 2018A Survey of Domain Adaptation for Neural Machine Translationcited by 64
  2. 2021Adapt-and-Distill: Developing Small, Fast and Effective Pretrained Language Models for Domainscited by 36
  3. 2021Pre-train or Annotate? Domain Adaptation with a Constrained Budgetcited by 27
  4. 2023Adapt in Contexts: Retrieval-Augmented Domain Adaptation via In-Context Learningcited by 16
  5. 2023RAPGen: An Approach for Fixing Code Inefficiencies in Zero-Shotcited by 14
  6. 2024LLMs in Biomedicine: A study on clinical Named Entity Recognitioncited by 12
  7. 2023Towards Anytime Fine-tuning: Continually Pre-trained Language Models with Hypernetwork Promptcited by 8
  8. 2023Generative User-Experience Research for Developing Domain-specific Natural Language Processing Applicationscited by 5
  9. 2024Automated Collection of Evaluation Dataset for Semantic Search in Low-Resource Domain Languagecited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 1.

3 papers.

  1. 2025Link Prediction for Event Logs in the Process Industrycited by 1
  2. 2025Contrastive Learning Using Graph Embeddings for Domain Adaptation of Language Models in the Process Industrycited by 0
  3. 2026Diagnosing Generalization Failures in Fine-Tuned LLMs: A Cross-Architectural Study on Phishing Detectioncited by 0