# On the Importance of Data Size in Probing Fine-tuned Models

arXiv: https://arxiv.org/abs/2203.09627 · HTML: https://mlsanity.com/paper/2203.09627/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 24 references that are in the map.
- First posted on arXiv 17 Mar 2022.

## Builds on (most cited first, 15 of 15)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference](https://mlsanity.com/paper/1704.05426/) (2017)
- [BERT Rediscovers the Classical NLP Pipeline](https://mlsanity.com/paper/1905.05950/) (2019)
- [What Does BERT Look At? An Analysis of BERT's Attention](https://mlsanity.com/paper/1906.04341/) (2019)
- [Neural Network Acceptability Judgments](https://mlsanity.com/paper/1805.12471/) (2018)
- [What do you learn from context? Probing for sentence structure in contextualized word representations](https://mlsanity.com/paper/1905.06316/) (2019)
- [Linguistic Knowledge and Transferability of Contextual Representations](https://mlsanity.com/paper/1903.08855/) (2019)
- [SentEval: An Evaluation Toolkit for Universal Sentence Representations](https://mlsanity.com/paper/1803.05449/) (2018)
- [What Happens To BERT Embeddings During Fine-tuning?](https://mlsanity.com/paper/2004.14448/) (2020)
- [Perturbed Masking: Parameter-free Probing for Analyzing and Interpreting BERT](https://mlsanity.com/paper/2004.14786/) (2020)
- [On the Interplay Between Fine-tuning and Sentence-level Probing for Linguistic Knowledge in Pre-trained Transformers](https://mlsanity.com/paper/2010.02616/) (2020)
- [Asking without Telling: Exploring Latent Ontologies in Contextual Representations](https://mlsanity.com/paper/2004.14513/) (2020)
- [How transfer learning impacts linguistic knowledge in deep NLP models?](https://mlsanity.com/paper/2105.15179/) (2021)
- [Spying on your neighbors: Fine-grained probing of contextual embeddings for information about surrounding words](https://mlsanity.com/paper/2005.01810/) (2020)

## Built on by (most cited first, 12 of 12)

- [On the Transformation of Latent Space in Fine-Tuned NLP Models](https://mlsanity.com/paper/2210.12696/) (2022)
- [The Power of LLM-Generated Synthetic Data for Stance Detection in Online Political Discussions](https://mlsanity.com/paper/2406.12480/) (2024)
- [Autoregressive Score Generation for Multi-trait Essay Scoring](https://mlsanity.com/paper/2403.08332/) (2024)
- [SQBC: Active Learning using LLM-Generated Synthetic Data for Stance Detection in Online Political Discussions](https://mlsanity.com/paper/2404.08078/) (2024)
- [An Empirical Study on the Transferability of Transformer Modules in Parameter-Efficient Fine-Tuning](https://mlsanity.com/paper/2302.00378/) (2023)
- [Is ChatGPT the Future of Causal Text Mining? A Comprehensive Evaluation and Analysis](https://mlsanity.com/paper/2402.14484/) (2024)
- [Pixology: Probing the Linguistic and Visual Capabilities of Pixel-based Language Models](https://mlsanity.com/paper/2410.12011/) (2024)
- [CR-COPEC: Causal Rationale of Corporate Performance Changes to Learn from Financial Reports](https://mlsanity.com/paper/2310.16095/) (2023)
- [BERT-Enhanced Retrieval Tool for Homework Plagiarism Detection System](https://mlsanity.com/paper/2404.01582/) (2024)
- [Automatic identification of diagnosis from hospital discharge letters via weakly supervised Natural Language Processing](https://mlsanity.com/paper/2410.15051/) (2024)
- [Optimizing Small Transformer-Based Language Models for Multi-Label Sentiment Analysis in Short Texts](https://mlsanity.com/paper/2509.04982/) (2025)
- [Supporting Workflow Reproducibility by Linking Bioinformatics Tools across Papers and Executable Code](https://mlsanity.com/paper/2603.08195/) (2026)

## Abstract (arXiv)

Several studies have investigated the reasons behind the effectiveness of fine-tuning, usually through the lens of probing. However, these studies often neglect the role of the size of the dataset on which the model is fine-tuned. In this paper, we highlight the importance of this factor and its undeniable role in probing performance. We show that the extent of encoded linguistic knowledge depends on the number of fine-tuning samples. The analysis also reveals that larger training data mainly affects higher layers, and that the extent of this change is a factor of the number of iterations updating the model during fine-tuning rather than the diversity of the training samples. Finally, we show through a set of experiments that fine-tuning data size affects the recoverability of the changes made to the model's linguistic knowledge.