mlsanityData to 7 Oct 2026

On the Importance of Data Size in Probing Fine-tuned Models

arXiv 2203.09627 ↗First posted 17 Mar 2022cs.CL cs.LGcited by 12

Not on the map. Why?

Abstract (arXiv)

Several studies have investigated the reasons behind the effectiveness of fine-tuning, usually through the lens of probing. However, these studies often neglect the role of the size of the dataset on which the model is fine-tuned. In this paper, we highlight the importance of this factor and its undeniable role in probing performance. We show that the extent of encoded linguistic knowledge depends on the number of fine-tuning samples. The analysis also reveals that larger training data mainly affects higher layers, and that the extent of this change is a factor of the number of iterations updating the model during fine-tuning rather than the diversity of the training samples. Finally, we show through a set of experiments that fine-tuning data size affects the recoverability of the changes made to the model's linguistic knowledge.

Key facts. Cited by 12 papers in the map (data through 7 Oct 2026). Builds on 15 of its 24 references that are in the map. First posted on arXiv 17 Mar 2022.

Builds on

15 of 24 references.

References · 15

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  3. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
  4. 2019BERT Rediscovers the Classical NLP Pipelinecited by 1,046
  5. 2019What Does BERT Look At? An Analysis of BERT's Attentioncited by 994
  6. 2018Neural Network Acceptability Judgmentscited by 952
  7. 2019What do you learn from context? Probing for sentence structure in contextualized word representationscited by 436
  8. 2019Linguistic Knowledge and Transferability of Contextual Representationscited by 382
  9. 2018SentEval: An Evaluation Toolkit for Universal Sentence Representationscited by 294
  10. 2020What Happens To BERT Embeddings During Fine-tuning?cited by 113
  11. 2020Perturbed Masking: Parameter-free Probing for Analyzing and Interpreting BERTcited by 109
  12. 2020On the Interplay Between Fine-tuning and Sentence-level Probing for Linguistic Knowledge in Pre-trained Transformerscited by 39
3 more
  1. 2020Asking without Telling: Exploring Latent Ontologies in Contextual Representationscited by 34
  2. 2021How transfer learning impacts linguistic knowledge in deep NLP models?cited by 32
  3. 2020Spying on your neighbors: Fine-grained probing of contextual embeddings for information about surrounding wordscited by 28

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 1; 2023: 2; 2024: 7; 2025: 1; 2026: 1.

12 papers.

  1. 2022On the Transformation of Latent Space in Fine-Tuned NLP Modelscited by 19
  2. 2024The Power of LLM-Generated Synthetic Data for Stance Detection in Online Political Discussionscited by 16
  3. 2024Autoregressive Score Generation for Multi-trait Essay Scoringcited by 8
  4. 2024SQBC: Active Learning using LLM-Generated Synthetic Data for Stance Detection in Online Political Discussionscited by 3
  5. 2023An Empirical Study on the Transferability of Transformer Modules in Parameter-Efficient Fine-Tuningcited by 2
  6. 2024Is ChatGPT the Future of Causal Text Mining? A Comprehensive Evaluation and Analysiscited by 2
  7. 2024Pixology: Probing the Linguistic and Visual Capabilities of Pixel-based Language Modelscited by 2
  8. 2023CR-COPEC: Causal Rationale of Corporate Performance Changes to Learn from Financial Reportscited by 1
  9. 2024BERT-Enhanced Retrieval Tool for Homework Plagiarism Detection Systemcited by 1
  10. 2024Automatic identification of diagnosis from hospital discharge letters via weakly supervised Natural Language Processingcited by 1
  11. 2025Optimizing Small Transformer-Based Language Models for Multi-Label Sentiment Analysis in Short Textscited by 0
  12. 2026Supporting Workflow Reproducibility by Linking Bioinformatics Tools across Papers and Executable Codecited by 0