Exploring Alignment in Shared Cross-lingual Spaces

Despite their remarkable ability to capture linguistic nuances across diverse languages, questions persist regarding the degree of alignment between languages in multilingual embeddings. Drawing inspiration from research on high-dimensional representations in neural language models, we employ clustering to uncover latent concepts within multilingual models. Our analysis focuses on quantifying the textitalignment and textitoverlap of these concepts across various languages within the latent space. To this end, we introduce two metrics CA and CO aimed at quantifying these aspects, enabling a deeper exploration of multilingual embeddings. Our study encompasses three multilingual models (textttmT5, textttmBERT, and textttXLM-R) and three downstream tasks (Machine Translation, Named Entity Recognition, and Sentiment Analysis). Key findings from our analysis include: i) deeper layers in the network demonstrate increased cross-lingual textitalignment due to the presence of language-agnostic concepts, ii) fine-tuning of the models enhances textitalignment within the latent space, and iii) such task-specific calibration helps in explaining the emergence of zero-shot capabilities in the models.footnoteThe code is available at urlhttps://github.com/baselmousi/multilingual-latent-concepts