mlsanityData to 7 Oct 2026

A Joint Matrix Factorization Analysis of Multilingual Representations

arXiv 2310.15513 ↗First posted 24 Oct 2023cs.CLcited by 5

Not on the map. Why?

Abstract (arXiv)

We present an analysis tool based on joint matrix factorization for comparing latent representations of multilingual and monolingual models. An alternative to probing, this tool allows us to analyze multiple sets of representations in a joint manner. Using this tool, we study to what extent and how morphosyntactic features are reflected in the representations learned by multilingual pre-trained models. We conduct a large-scale empirical study of over 33 languages and 17 morphosyntactic categories. Our findings demonstrate variations in the encoding of morphosyntactic information across upper and lower layers, with category-specific differences influenced by language properties. Hierarchical clustering of the factorization outputs yields a tree structure that is related to phylogenetic trees manually crafted by linguists. Moreover, we find the factorization outputs exhibit strong associations with performance observed across different cross-lingual tasks. We release our code to facilitate future research.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Builds on 41 of its 62 references that are in the map. First posted on arXiv 24 Oct 2023.

Builds on

41 of 62 references.

References · 41

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  4. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
  5. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,973
  6. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,813
  7. 2019Similarity of Neural Network Representations Revisitedcited by 1,695
  8. 2019Cross-lingual Language Model Pretrainingcited by 1,088
  9. 2019BERT Rediscovers the Classical NLP Pipelinecited by 1,046
  10. 2018XNLI: Evaluating Cross-lingual Sentence Representationscited by 913
  11. 2021Probing Classifiers: Promises, Shortcomings, and Advancescited by 831
  12. 2020A Primer in BERTology: What we know about how BERT workscited by 807
29 more
  1. 2020XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalizationcited by 738
  2. 2019On the Cross-lingual Transferability of Monolingual Representationscited by 575
  3. 2020TyDi QA: A Benchmark for Information-Seeking Question Answering in Typologically Diverse Languagescited by 486
  4. 2018Massively Multilingual Sentence Embeddings for Zero-Shot Cross-Lingual Transfer and Beyondcited by 434
  5. 2019MLQA: Evaluating Cross-lingual Extractive Question Answeringcited by 339
  6. 2018Insights on representational similarity in neural networks with canonical correlationcited by 318
  7. 2019PAWS-X: A Cross-lingual Adversarial Dataset for Paraphrase Identificationcited by 262
  8. 2019Massively Multilingual Neural Machine Translation in the Wild: Findings and Challengescited by 253
  9. 2019Massively Multilingual Neural Machine Translationcited by 246
  10. 2020Information-Theoretic Probing for Linguistic Structurecited by 204
  11. 2019Cross-Lingual Ability of Multilingual BERT: An Empirical Studycited by 182
  12. 2020What Happens To BERT Embeddings During Fine-tuning?cited by 113
  13. 2016TensorLy: Tensor Learning in Pythoncited by 104
  14. 2019Investigating Multilingual NMT Representations at Scalecited by 68
  15. 2021First Align, then Predict: Understanding the Cross-Lingual Ability of Multilingual BERTcited by 64
  16. 2019How Language-Neutral is Multilingual BERT?cited by 51
  17. 2023Detecting and Mitigating Hallucinations in Multilingual Summarisationcited by 48
  18. 2020Intrinsic Probing through Dimension Selectioncited by 45
  19. 2018UniMorph 2.0: Universal Morphologycited by 40
  20. 2022Gold Doesn't Always Glitter: Spectral Removal of Linear and Nonlinear Guarded Attribute Informationcited by 33
  21. 2022Discovering Language-neutral Sub-networks in Multilingual Language Modelscited by 28
  22. 2022Match the Script, Adapt if Multilingual: Analyzing the Effect of Multilingual Pretraining on Cross-lingual Transferabilitycited by 26
  23. 2020What does it mean to be language-agnostic? Probing multilingual sentence encoders for typological propertiescited by 23
  24. 2023Efficient fair PCA for fair representation learningcited by 23
  25. 2018Convex Formulations for Fair Principal Component Analysiscited by 20
  26. 2019Multilingual Graphemic Hybrid ASR with Massive Data Augmentationcited by 13
  27. 2018Marrying Universal Dependencies and Universal Morphologycited by 11
  28. 2021Locating Language-Specific Information in Contextualized Embeddingscited by 6
  29. 2022Understanding Domain Learning in Language Models Through Subpopulation Analysiscited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 1; 2026: 2.

5 papers.

  1. 2024Spectral Editing of Activations for Large Language Model Alignmentcited by 54
  2. 2024Layer by Layer: Uncovering Where Multi-Task Learning Happens in Instruction-Tuned Large Language Modelscited by 15
  3. 2026Can Embedding Similarity Predict Cross-Lingual Transfer? A Systematic Study on African Languagescited by 3
  4. 2026Differentiable Faithfulness Alignment for Cross-Model Circuit Transfercited by 3
  5. 2025Iterative Multilingual Spectral Attribute Erasurecited by 1