# Exploring Italian sentence embeddings properties through multi-tasking

arXiv: https://arxiv.org/abs/2409.06622 · HTML: https://mlsanity.com/paper/2409.06622/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 10 of its 19 references that are in the map.
- First posted on arXiv 10 Sep 2024.

## Builds on (most cited first, 10 of 10)

- [An Overview of Multi-Task Learning in Deep Neural Networks](https://mlsanity.com/paper/1706.05098/) (2017)
- [Which Tasks Should Be Learned Together in Multi-task Learning?](https://mlsanity.com/paper/1905.07553/) (2019)
- [Multi-Task Learning in Natural Language Processing: An Overview](https://mlsanity.com/paper/2109.09138/) (2021)
- [Multi-task Learning for Multilingual Neural Machine Translation](https://mlsanity.com/paper/2010.02523/) (2020)
- [A Survey of Multi-task Learning in Natural Language Processing: Regarding Task Relatedness and Training Methods](https://mlsanity.com/paper/2204.03508/) (2022)
- [Verb Argument Structure Alternations in Word and Sentence Embeddings](https://mlsanity.com/paper/1811.10773/) (2018)
- [Blackbird language matrices (BLM), a new task for rule-like generalization in neural networks: Motivations and Formal Specifications](https://mlsanity.com/paper/2306.11444/) (2023)
- [Are there identifiable structural parts in the sentence embedding whole?](https://mlsanity.com/paper/2406.16563/) (2024)
- [MOCHA: A Multi-Task Training Approach for Coherent Text Generation from Cognitive Perspective](https://mlsanity.com/paper/2210.14650/) (2022)
- [Grammatical information in BERT sentence embeddings as two-dimensional arrays](https://mlsanity.com/paper/2312.09890/) (2023)

## Built on by (most cited first, 6 of 6)

- [Sentence Smith: Controllable Edits for Evaluating Text Embeddings](https://mlsanity.com/paper/2502.14734/) (2025)
- [Modelling the Morphology of Verbal Paradigms: A Case Study in the Tokenization of Turkish and Hebrew](https://mlsanity.com/paper/2602.05648/) (2026)
- [Datasets for Verb Alternations across Languages: BLM Templates and Data Augmentation Strategies](https://mlsanity.com/paper/2603.15295/) (2026)
- [ALEE: Any-Language Evaluation of Embeddings via English-Centric Minimal Pairs](https://mlsanity.com/paper/2607.00171/) (2026)
- [Blackbird Language Matrices: A Framework to Investigate the Linguistic Competence of Language Models](https://mlsanity.com/paper/2602.20966/) (2026)
- [Comparing Natural and Synthetic Structured Data: A Study of the Passive Verb Alternation in French and Italian](https://mlsanity.com/paper/2603.25227/) (2026)

## Abstract (arXiv)

We investigate to what degree existing LLMs encode abstract linguistic information in Italian in a multi-task setting. We exploit curated synthetic data on a large scale -- several Blackbird Language Matrices (BLMs) problems in Italian -- and use them to study how sentence representations built using pre-trained language models encode specific syntactic and semantic information. We use a two-level architecture to model separately a compression of the sentence embeddings into a representation that contains relevant information for a task, and a BLM task. We then investigate whether we can obtain compressed sentence representations that encode syntactic and semantic information relevant to several BLM tasks. While we expected that the sentence structure -- in terms of sequence of phrases/chunks -- and chunk properties could be shared across tasks, performance and error analysis show that the clues for the different tasks are encoded in different manners in the sentence embeddings, suggesting that abstract linguistic notions such as constituents or thematic roles does not seem to be present in the pretrained sentence embeddings.