mlsanityData to 14 Oct 2026

Contrastive Learning Using Graph Embeddings for Domain Adaptation of Language Models in the Process Industry

arXiv 2510.04631 ↗First posted 6 Oct 2025cs.CL cs.IRcited by 0

Not on the map. Why?

Abstract (arXiv)

Recent trends in NLP utilize knowledge graphs (KGs) to enhance pretrained language models by incorporating additional knowledge from the graph structures to learn domain-specific terminology or relationships between documents that might otherwise be overlooked. This paper explores how SciNCL, a graph-aware neighborhood contrastive learning methodology originally designed for scientific publications, can be applied to the process industry domain, where text logs contain crucial information about daily operations and are often structured as sparse KGs. Our experiments demonstrate that language models fine-tuned with triplets derived from graph embeddings (GE) outperform a state-of-the-art mE5-large text encoder by 9.8-14.3% (5.45-7.96p) on the proprietary process industry text embedding benchmark (PITEB) while having 3 times fewer parameters.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 36 of its 62 references that are in the map. First posted on arXiv 6 Oct 2025.

Builds on

36 of 62 references.

References · 36

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  3. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  4. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,935
  5. 2015FaceNet: A Unified Embedding for Face Recognition and Clusteringcited by 2,849
  6. 2017Billion-scale similarity search with GPUscited by 2,497
  7. 2019BioBERT: a pre-trained biomedical language representation model for biomedical text miningcited by 1,947
  8. 2016MS MARCO: A Human Generated MAchine Reading COmprehension Datasetcited by 1,605
  9. 2020Don't Stop Pretraining: Adapt Language Models to Domains and Taskscited by 1,565
  10. 2019SciBERT: A Pretrained Language Model for Scientific Textcited by 1,414
  11. 2020Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hyperspherecited by 1,405
  12. 2021BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Modelscited by 1,252
24 more
  1. 2024M3-Embedding: Multi-Linguality, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillationcited by 1,073
  2. 2019A Theoretical Analysis of Contrastive Unsupervised Representation Learningcited by 530
  3. 2019ERNIE: Enhanced Language Representation with Informative Entitiescited by 528
  4. 2020Pre-trained Models for Natural Language Processing: A Surveycited by 505
  5. 2019ClinicalBERT: Modeling Clinical Notes and Predicting Hospital Readmissioncited by 493
  6. 2020SPECTER: Document-level Representation Learning using Citation-informed Transformerscited by 430
  7. 2024Multilingual E5 Text Embeddings: A Technical Reportcited by 372
  8. 2019KEPLER: A Unified Model for Knowledge Embedding and Pre-trained Language Representationcited by 356
  9. 2019K-BERT: Enabling Language Representation with Knowledge Graphcited by 304
  10. 2019Knowledge Enhanced Contextual Word Representationscited by 296
  11. 2017Sampling Matters in Deep Embedding Learningcited by 285
  12. 2024mGTE: Generalized Long-Context Text Representation and Reranking Models for Multilingual Text Retrievalcited by 236
  13. 2022LinkBERT: Pretraining Language Models with Document Linkscited by 211
  14. 2020German's Next Language Modelcited by 144
  15. 2019PyTorch-BigGraph: A Large-scale Graph Embedding Systemcited by 119
  16. 2022Neighborhood Contrastive Learning for Scientific Document Representations with Citation Embeddingscited by 75
  17. 2022TwHIN-BERT: A Socially-Enriched Pre-trained Language Model for Multilingual Tweet Representations at Twittercited by 45
  18. 2025Training Sparse Mixture Of Experts Text Embedding Modelscited by 32
  19. 2016Hard Negative Mining for Metric Learning Based Zero-Shot Classificationcited by 18
  20. 2020MaintNet: A Collaborative Open-Source Library for Predictive Maintenance Language Resourcescited by 8
  21. 2023Generative User-Experience Research for Developing Domain-specific Natural Language Processing Applicationscited by 5
  22. 2024Automated Collection of Evaluation Dataset for Semantic Search in Low-Resource Domain Languagecited by 4
  23. 2025Efficient Domain-adaptive Continual Pretraining for the Process Industry in the German Languagecited by 3
  24. 2025Link Prediction for Event Logs in the Process Industrycited by 1

Built on by

Not cited yet.