mlsanityData to 14 Oct 2026

Continued Pretraining of FinBERT on Finnish Histopathological Reports: Train-Time Signals and Proxy Downstream Correlations

arXiv 2604.14815 ↗First posted 16 Apr 2026cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

In Natural Language Processing (NLP) classification tasks where a lack of labeled data is an issue, continued pretraining (CPT) of transformer models on unlabeled data is an established approach. In this paper, we have two aims. (1) We describe our observations from continued pretraining of the Finnish BERT transformer model (FinBERT) on a Finnish histopathological dataset (below, the Histopathology data). (2) Since the Histopathology data has no classification labels, we gather public Finnish datasets as proxy data to analyze whether the signals observed in (1) are associated with downstream classification gains. We observe that CPT train-time loss curves differ strongly by domain, and that, in an exploratory analysis, certain CPT-derived features correlate with proxy classification improvement. In particular, this report contributes to the limited literature on NLP for Finnish healthcare data.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 28 of its 58 references that are in the map. First posted on arXiv 16 Apr 2026.

Builds on

28 of 58 references.

References · 28

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  4. 2020Scaling Laws for Neural Language Modelscited by 6,586
  5. 2022Training Compute-Optimal Large Language Modelscited by 3,019
  6. 2018Deep contextualized word representationscited by 2,290
  7. 2019BioBERT: a pre-trained biomedical language representation model for biomedical text miningcited by 1,947
  8. 2019Similarity of Neural Network Representations Revisitedcited by 1,755
  9. 2020Don't Stop Pretraining: Adapt Language Models to Domains and Taskscited by 1,565
  10. 2019Publicly Available Clinical BERT Embeddingscited by 1,002
  11. 2020A Primer in BERTology: What we know about how BERT workscited by 815
  12. 2024Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder for Fast, Memory Efficient, and Long Context Finetuning and Inferencecited by 554
16 more
  1. 2018Insights on representational similarity in neural networks with canonical correlationcited by 324
  2. 2021Scaling Laws for Transfercited by 272
  3. 2017All-but-the-Top: Simple and Effective Postprocessing for Word Representationscited by 212
  4. 2015Semi-supervised Sequence Learningcited by 198
  5. 2019Multilingual is not enough: BERT for Finnishcited by 119
  6. 2021Generalized Shape Metrics on Neural Representationscited by 112
  7. 2017Offline bilingual word vectors, orthogonal transformations and the inverted softmaxcited by 96
  8. 2019Representation Similarity Analysis for Efficient Task taxonomy & Transfer Learningcited by 86
  9. 2020Classifying the classifier: dissecting the weight space of neural networkscited by 65
  10. 2019Universality and individuality in neural dynamics across large populations of recurrent networkscited by 61
  11. 2021IsoScore: Measuring the Uniformity of Embedding Space Utilizationcited by 53
  12. 2024D-CPT Law: Domain-specific Continual Pre-Training Scaling Law for Large Language Modelscited by 41
  13. 2021Fine-Tuned Transformers Show Clusters of Similar Representations Across Layerscited by 30
  14. 2024Loss-to-Loss Prediction: Scaling Laws for All Datasetscited by 15
  15. 2024Evaluating Representational Similarity Measures from the Lens of Functional Correspondencecited by 12
  16. 2025Measuring the Measures: Discriminative Capacity of Representational Similarity Metrics Across Model Familiescited by 2

Built on by

Not cited yet.