mlsanityData to 14 Oct 2026

Domain Pre-training Impact on Representations

arXiv 2505.24455 ↗First posted 30 May 2025cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

This empirical study analyzes the effects of the pre-training corpus on the quality of learned transformer representations. We focus on the representation quality induced solely through pre-training. Our experiments show that pre-training on a small, specialized corpus can yield effective representations, and that the success of combining a generic and a specialized corpus depends on the distributional similarity between the target task and the specialized corpus.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 34 of its 56 references that are in the map. First posted on arXiv 30 May 2025.

Builds on

34 of 56 references.

References · 34

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  3. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  4. 2015Character-level Convolutional Networks for Text Classificationcited by 2,330
  5. 2018Deep contextualized word representationscited by 2,290
  6. 2019BioBERT: a pre-trained biomedical language representation model for biomedical text miningcited by 1,947
  7. 2020Don't Stop Pretraining: Adapt Language Models to Domains and Taskscited by 1,565
  8. 2019SciBERT: A Pretrained Language Model for Scientific Textcited by 1,414
  9. 2020Domain-Specific Language Model Pretraining for Biomedical Natural Language Processingcited by 1,009
  10. 2020A Primer in BERTology: What we know about how BERT workscited by 815
  11. 2019How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddingscited by 784
  12. 2018What you can cram into a single vector: Probing sentence embeddings for linguistic propertiescited by 457
22 more
  1. 2019Nuanced Metrics for Measuring Unintended Bias with Real Data for Text Classificationcited by 404
  2. 2019Linguistic Knowledge and Transferability of Contextual Representationscited by 383
  3. 2020BERTweet: A pre-trained language model for English Tweetscited by 346
  4. 2016Fine-grained Analysis of Sentence Embeddings Using Auxiliary Prediction Taskscited by 216
  5. 2020Few-Shot Learning via Learning the Representation, Provablycited by 214
  6. 2016Ex Machina: Personal Attacks Seen at Scalecited by 210
  7. 2020HateBERT: Retraining BERT for Abusive Language Detection in Englishcited by 207
  8. 2020On the Theory of Transfer Learning: The Importance of Task Diversitycited by 186
  9. 2019BERT Post-Training for Review Reading Comprehension and Aspect-based Sentiment Analysiscited by 176
  10. 2020Unsupervised Domain Clusters in Pretrained Language Modelscited by 160
  11. 2020When Do You Need Billions of Words of Pretraining Data?cited by 104
  12. 2021Probing Across Time: What Does RoBERTa Know and When?cited by 73
  13. 2024Probing the Emergence of Cross-lingual Alignment during LLM Trainingcited by 39
  14. 2022Analyzing the Mono- and Cross-Lingual Pretraining Dynamics of Multilingual Language Modelscited by 32
  15. 2020Pretrained Language Model Embryology: The Birth of ALBERTcited by 30
  16. 2021DirectProbe: Studying Representations without Classifierscited by 25
  17. 2023On the Provable Advantage of Unsupervised Pretrainingcited by 22
  18. 2022Downstream Datasets Make Surprisingly Good Pretraining Corporacited by 20
  19. 2022Pre-trained Language Models for Keyphrase Generation: A Thorough Empirical Studycited by 13
  20. 2024On the Generalization Ability of Unsupervised Pretrainingcited by 8
  21. 2021Comparing Text Representations: A Theory-Driven Approachcited by 5
  22. 2023Analyzing Text Representations by Measuring Task Alignmentcited by 2

Built on by

Not cited yet.