# Deep Clustering of Text Representations for Supervision-free Probing of Syntax

arXiv: https://arxiv.org/abs/2010.12784 · HTML: https://mlsanity.com/paper/2010.12784/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 24 Oct 2020.

## Built on by (most cited first, 7 of 7)

- [On Behalf of the Stakeholders: Trends in NLP Model Interpretability in the Era of LLMs](https://mlsanity.com/paper/2407.19200/) (2024)
- [Beyond Prompting: Making Pre-trained Language Models Better Zero-shot Learners by Clustering Representations](https://mlsanity.com/paper/2210.16637/) (2022)
- [Scaling up Discovery of Latent Concepts in Deep NLP Models](https://mlsanity.com/paper/2308.10263/) (2023)
- [Linguistic Interpretability of Transformer-based Language Models: a systematic review](https://mlsanity.com/paper/2504.08001/) (2025)
- [Masked Part-Of-Speech Model: Does Modeling Long Context Help Unsupervised POS-tagging?](https://mlsanity.com/paper/2206.14969/) (2022)
- [Forecasting Auxiliary Energy Consumption for Electric Heavy-Duty Vehicles](https://mlsanity.com/paper/2311.16003/) (2023)
- [Representation of syntax in LLMs through the lens of linear distance and similarity-aware entropy](https://mlsanity.com/paper/2608.27813/) (2026)

## Abstract (arXiv)

We explore deep clustering of text representations for unsupervised model interpretation and induction of syntax. As these representations are high-dimensional, out-of-the-box methods like KMeans do not work well. Thus, our approach jointly transforms the representations into a lower-dimensional cluster-friendly space and clusters them. We consider two notions of syntax: Part of speech Induction (POSI) and constituency labelling (CoLab) in this work. Interestingly, we find that Multilingual BERT (mBERT) contains surprising amount of syntactic knowledge of English; possibly even as much as English BERT (EBERT). Our model can be used as a supervision-free probe which is arguably a less-biased way of probing. We find that unsupervised probes show benefits from higher layers as compared to supervised probes. We further note that our unsupervised probe utilizes EBERT and mBERT representations differently, especially for POSI. We validate the efficacy of our probe by demonstrating its capabilities as an unsupervised syntax induction technique. Our probe works well for both syntactic formalisms by simply adapting the input representations. We report competitive performance of our probe on 45-tag English POSI, state-of-the-art performance on 12-tag POSI across 10 languages, and competitive results on CoLab. We also perform zero-shot syntax induction on resource impoverished languages and report strong results.