mlsanityData to 14 Oct 2026

MOSAIC: Masked Objective with Selective Adaptation for In-domain Contrastive Learning

arXiv 2510.16797 ↗First posted 19 Oct 2025cs.CL cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

We introduce MOSAIC (Masked Objective with Selective Adaptation for In-domain Contrastive learning), a multi-stage framework for domain adaptation of text embedding models that incorporates joint domain-specific masked supervision. Our approach addresses the challenges of adapting large-scale general-domain text embedding models to specialized domains. By jointly optimizing masked language modeling (MLM) and contrastive objectives within a unified training pipeline, our method enables effective learning of domain-relevant representations while preserving the robust semantic discrimination properties of the original model. We empirically validate our approach on both high-resource and low-resource domains, achieving improvements up to 13.4% in NDCG@10 (Normalized Discounted Cumulative Gain) over strong general-domain baselines. Comprehensive ablation studies further demonstrate the effectiveness of each component, highlighting the importance of balanced joint supervision and staged adaptation.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 42 of its 56 references that are in the map. First posted on arXiv 19 Oct 2025.

Builds on

42 of 56 references.

References · 42

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  4. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  5. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  6. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  7. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  8. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,307
  9. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,935
  10. 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,173
  11. 2019BioBERT: a pre-trained biomedical language representation model for biomedical text miningcited by 1,947
  12. 2019SciBERT: A Pretrained Language Model for Scientific Textcited by 1,414
30 more
  1. 2018Learning deep representations by mutual information estimation and maximizationcited by 1,273
  2. 2022Text Embeddings by Weakly-Supervised Contrastive Pre-trainingcited by 1,253
  3. 2020Domain-Specific Language Model Pretraining for Biomedical Natural Language Processingcited by 1,009
  4. 2019Publicly Available Clinical BERT Embeddingscited by 1,002
  5. 2017SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluationcited by 1,001
  6. 2019How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddingscited by 784
  7. 2019Learning Representations by Maximizing Mutual Information Across Viewscited by 734
  8. 2023Towards General Text Embeddings with Multi-stage Contrastive Learningcited by 723
  9. 2023C-Pack: Packed Resources For General Chinese Embeddingscited by 578
  10. 2020DeCLUTR: Deep Contrastive Learning for Unsupervised Textual Representationscited by 251
  11. 2020InfoXLM: An Information-Theoretic Framework for Cross-Lingual Language Model Pre-Trainingcited by 237
  12. 2020CLEAR: Contrastive Learning for Sentence Representationcited by 186
  13. 2020Augmented SBERT: Data Augmentation Method for Improving Bi-Encoders for Pairwise Sentence Scoring Taskscited by 138
  14. 2021COCO-LM: Correcting and Contrasting Text Sequences for Language Model Pretrainingcited by 126
  15. 2021GPL: Generative Pseudo Labeling for Unsupervised Domain Adaptation of Dense Retrievalcited by 119
  16. 2021TSDAE: Using Transformer-based Sequential Denoising Auto-Encoder for Unsupervised Sentence Embedding Learningcited by 116
  17. 2021Fast, Effective, and Self-Supervised: Transforming Masked Language Models into Universal Lexical and Sentence Encoderscited by 88
  18. 2019A Mutual Information Maximization Perspective of Language Representation Learningcited by 85
  19. 2024Arctic-Embed: Scalable, Efficient, and Accurate Text Embedding Modelscited by 61
  20. 2020Train No Evil: Selective Masking for Task-Guided Pre-Trainingcited by 36
  21. 2021Efficient Domain Adaptation of Language Models via Adaptive Tokenizationcited by 34
  22. 2022InfoCSE: Information-aggregated Contrastive Learning of Sentence Embeddingscited by 26
  23. 2020Inexpensive Domain Adaptation of Pretrained Language Models: Case Studies on Biomedical NER and Covid-19 QAcited by 23
  24. 2022InforMask: Unsupervised Informative Masking for Language Model Pretrainingcited by 15
  25. 2023Efficient Domain Adaptation of Sentence Embeddings Using Adapterscited by 6
  26. 2023Leveraging Domain Adaptation and Data Augmentation to Improve Qur'anic IR in English and Arabiccited by 4
  27. 2025Multi-stage Training of Bilingual Islamic LLM for Neural Passage Retrievalcited by 4
  28. 2024Building an Efficient Multilingual Non-Profit IR System for the Islamic Domain Leveraging Multiprocessing Design in Rustcited by 3
  29. 2024Language Model Adaptation to Specialized Domains through Selective Masking based on Genre and Topical Characteristicscited by 2
  30. 2025Efficient and Versatile Model for Multilingual Information Retrieval of Islamic Text: Development and Deployment in Real-World Scenarioscited by 1

Built on by

Not cited yet.