# LINSPECTOR: Multilingual Probing Tasks for Word Representations

arXiv: https://arxiv.org/abs/1903.09442 · HTML: https://mlsanity.com/paper/1903.09442/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 22 Mar 2019.

## Built on by (most cited first, 22 of 22)

- [RuBLiMP: Russian Benchmark of Linguistic Minimal Pairs](https://mlsanity.com/paper/2406.19232/) (2024)
- [Multilingual Pretraining for Pixel Language Models](https://mlsanity.com/paper/2505.21265/) (2025)
- [Shaking Syntactic Trees on the Sesame Street: Multilingual Probing with Controllable Perturbations](https://mlsanity.com/paper/2109.14017/) (2021)
- [Interpretation of Black Box NLP Models: A Survey](https://mlsanity.com/paper/2203.17081/) (2022)
- [Probing LLMs for Joint Encoding of Linguistic Categories](https://mlsanity.com/paper/2310.18696/) (2023)
- [A multilabel approach to morphosyntactic probing](https://mlsanity.com/paper/2104.08464/) (2021)
- [Morphosyntactic probing of multilingual BERT models](https://mlsanity.com/paper/2306.06205/) (2023)
- [Code-Mixed Probes Show How Pre-Trained Models Generalise On Code-Switched Text](https://mlsanity.com/paper/2403.04872/) (2024)
- [Evaluation Benchmarks for Spanish Sentence Representations](https://mlsanity.com/paper/2204.07571/) (2022)
- [Morph Call: Probing Morphosyntactic Content of Multilingual Transformers](https://mlsanity.com/paper/2104.12847/) (2021)
- [A Latent-Variable Model for Intrinsic Probing](https://mlsanity.com/paper/2201.08214/) (2022)
- [RuSentEval: Linguistic Source, Encoder Force!](https://mlsanity.com/paper/2103.00573/) (2021)
- [Probing with Noise: Unpicking the Warp and Weft of Embeddings](https://mlsanity.com/paper/2210.12206/) (2022)
- [Universal and Independent: Multilingual Probing Framework for Exhaustive Model Interpretation and Evaluation](https://mlsanity.com/paper/2210.13236/) (2022)
- [PARADISE: Evaluating Implicit Planning Skills of Language Models with Procedural Warnings and Tips Dataset](https://mlsanity.com/paper/2403.03167/) (2024)
- [Russian SuperGLUE 1.1: Revising the Lessons not Learned by Russian NLP models](https://mlsanity.com/paper/2202.07791/) (2022)
- [Pixology: Probing the Linguistic and Visual Capabilities of Pixel-based Language Models](https://mlsanity.com/paper/2410.12011/) (2024)
- [Sentence Ambiguity, Grammaticality and Complexity Probes](https://mlsanity.com/paper/2210.06928/) (2022)
- [Implicit Priors Editing in Stable Diffusion via Targeted Token Adjustment](https://mlsanity.com/paper/2412.03400/) (2024)
- [Why Do Neural Language Models Still Need Commonsense Knowledge to Handle Semantic Variations in Question Answering?](https://mlsanity.com/paper/2209.00599/) (2022)
- [Assessing Linguistic Generalisation in Language Models: A Dataset for Brazilian Portuguese](https://mlsanity.com/paper/2305.14070/) (2023)
- [Exposure is Optional: Learning Unlike Coordination in Language Models](https://mlsanity.com/paper/2607.20251/) (2026)

## Abstract (arXiv)

Despite an ever growing number of word representation models introduced for a large number of languages, there is a lack of a standardized technique to provide insights into what is captured by these models. Such insights would help the community to get an estimate of the downstream task performance, as well as to design more informed neural architectures, while avoiding extensive experimentation which requires substantial computational resources not all researchers have access to. A recent development in NLP is to use simple classification tasks, also called probing tasks, that test for a single linguistic feature such as part-of-speech. Existing studies mostly focus on exploring the linguistic information encoded by the continuous representations of English text. However, from a typological perspective the morphologically poor English is rather an outlier: the information encoded by the word order and function words in English is often stored on a morphological level in other languages. To address this, we introduce 15 type-level probing tasks such as case marking, possession, word length, morphological tag count and pseudoword identification for 24 languages. We present a reusable methodology for creation and evaluation of such tests in a multilingual setting. We then present experiments on several diverse multilingual word embedding models, in which we relate the probing task performance for a diverse set of languages to a range of five classic NLP tasks: POS-tagging, dependency parsing, semantic role labeling, named entity recognition and natural language inference. We find that a number of probing tests have significantly high positive correlation to the downstream tasks, especially for morphologically rich languages. We show that our tests can be used to explore word embeddings or black-box neural models for linguistic cues in a multilingual setting.