# Linguistic Profiling of a Neural Language Model

arXiv: https://arxiv.org/abs/2010.01869 · HTML: https://mlsanity.com/paper/2010.01869/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 5 Oct 2020.

## Built on by (most cited first, 23 of 23)

- [Towards Measuring and Modeling "Culture" in LLMs: A Survey](https://mlsanity.com/paper/2403.15412/) (2024)
- [Language Model Behavior: A Comprehensive Survey](https://mlsanity.com/paper/2303.11504/) (2023)
- [AttentionViz: A Global View of Transformer Attention](https://mlsanity.com/paper/2305.03210/) (2023)
- [How is BERT surprised? Layerwise detection of linguistic anomalies](https://mlsanity.com/paper/2105.07452/) (2021)
- [Acceptability Judgements via Examining the Topology of Attention Maps](https://mlsanity.com/paper/2205.09630/) (2022)
- [Incorporating Distributions of Discourse Structure for Long Document Abstractive Summarization](https://mlsanity.com/paper/2305.16784/) (2023)
- [Predicting Fine-Tuning Performance with Probing](https://mlsanity.com/paper/2210.07352/) (2022)
- [RST-LoRA: A Discourse-Aware Low-Rank Adaptation for Long Document Abstractive Summarization](https://mlsanity.com/paper/2405.00657/) (2024)
- [Shaking Syntactic Trees on the Sesame Street: Multilingual Probing with Controllable Perturbations](https://mlsanity.com/paper/2109.14017/) (2021)
- [Stress-testing Machine Generated Text Detection: Shifting Language Models Writing Style to Fool Detectors](https://mlsanity.com/paper/2505.24523/) (2025)
- [Linguistic Interpretability of Transformer-based Language Models: a systematic review](https://mlsanity.com/paper/2504.08001/) (2025)
- [On the Interplay Between Fine-tuning and Composition in Transformers](https://mlsanity.com/paper/2105.14668/) (2021)
- [Morph Call: Probing Morphosyntactic Content of Multilingual Transformers](https://mlsanity.com/paper/2104.12847/) (2021)
- [Can BERT eat RuCoLA? Topological Data Analysis to Explain](https://mlsanity.com/paper/2304.01680/) (2023)
- [Lacking the embedding of a word? Look it up into a traditional dictionary](https://mlsanity.com/paper/2109.11763/) (2021)
- [Establishing Trustworthiness: Rethinking Tasks and Model Evaluation](https://mlsanity.com/paper/2310.05442/) (2023)
- [Assessing the Use of Prosody in Constituency Parsing of Imperfect Transcripts](https://mlsanity.com/paper/2106.07794/) (2021)
- [Taxonomy of Abstractive Dialogue Summarization: Scenarios, Approaches and Future Directions](https://mlsanity.com/paper/2210.09894/) (2022)
- [The Grammar of Transformers: A Systematic Review of Interpretability Research on Syntactic Knowledge in Language Models](https://mlsanity.com/paper/2601.19926/) (2026)
- [To Understand Representation of Layer-aware Sequence Encoders as Multi-order-graph](https://mlsanity.com/paper/2101.06397/) (2021)
- [Linguistic Knowledge Can Enhance Encoder-Decoder Models (If You Let It)](https://mlsanity.com/paper/2402.17608/) (2024)
- [Evaluating Self-Generated Documents for Enhancing Retrieval-Augmented Generation with Large Language Models](https://mlsanity.com/paper/2410.13192/) (2024)
- [Making Sense of Korean Sentences: A Comprehensive Evaluation of LLMs through KoSEnd Dataset](https://mlsanity.com/paper/2507.03378/) (2025)

## Abstract (arXiv)

In this paper we investigate the linguistic knowledge learned by a Neural Language Model (NLM) before and after a fine-tuning process and how this knowledge affects its predictions during several classification problems. We use a wide set of probing tasks, each of which corresponds to a distinct sentence-level feature extracted from different levels of linguistic annotation. We show that BERT is able to encode a wide range of linguistic characteristics, but it tends to lose this information when trained on specific downstream tasks. We also find that BERT's capacity to encode different kind of linguistic properties has a positive influence on its predictions: the more it stores readable linguistic information of a sentence, the higher will be its capacity of predicting the expected label assigned to that sentence.