mlsanityData to 7 Oct 2026

Do Multi-Lingual Pre-trained Language Models Reveal Consistent Token Attributions in Different Languages?

arXiv 2112.12356 ↗First posted 23 Dec 2021cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

During the past several years, a surge of multi-lingual Pre-trained Language Models (PLMs) has been proposed to achieve state-of-the-art performance in many cross-lingual downstream tasks. However, the understanding of why multi-lingual PLMs perform well is still an open domain. For example, it is unclear whether multi-Lingual PLMs reveal consistent token attributions in different languages. To address this, in this paper, we propose a Cross-lingual Consistency of Token Attributions (CCTA) evaluation framework. Extensive experiments in three downstream tasks demonstrate that multi-lingual PLMs assign significantly different attributions to multi-lingual synonyms. Moreover, we have the following observations: 1) the Spanish achieves the most consistent token attributions in different languages when it is used for training PLMs; 2) the consistency of token attributions strongly correlates with performance in downstream tasks.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 29 of its 33 references that are in the map. First posted on arXiv 23 Dec 2021.

Builds on

29 of 33 references.

References · 29

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
  3. 2017Axiomatic Attribution for Deep Networkscited by 3,350
  4. 2018Deep contextualized word representationscited by 2,285
  5. 2019Cross-lingual Language Model Pretrainingcited by 1,088
  6. 2019BERT Rediscovers the Classical NLP Pipelinecited by 1,046
  7. 2020XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalizationcited by 738
  8. 2019How multilingual is Multilingual BERT?cited by 642
  9. 2019Linguistic Knowledge and Transferability of Contextual Representationscited by 382
  10. 2019Beto, Bentz, Becas: The Surprising Cross-Lingual Effectiveness of BERTcited by 301
  11. 2019Cross-Lingual Ability of Multilingual BERT: An Empirical Studycited by 182
  12. 2020Probing Pretrained Language Models for Lexical Semanticscited by 156
17 more
  1. 2020Finding Universal Grammatical Relations in Multilingual BERTcited by 108
  2. 2020On the Language Neutrality of Pre-trained Multilingual Representationscited by 82
  3. 2019Investigating Multilingual NMT Representations at Scalecited by 68
  4. 2018On the Limitations of Unsupervised Bilingual Dictionary Inductioncited by 61
  5. 2019How Language-Neutral is Multilingual BERT?cited by 51
  6. 2020Are All Good Word Vector Spaces Isomorphic?cited by 36
  7. 2019Bilingual Lexicon Induction with Semi-supervision in Non-Isometric Embedding Spacescited by 35
  8. 2020What's so special about BERT's layers? A closer look at the NLP pipeline in monolingual and multilingual modelscited by 35
  9. 2020Do Neural Language Models Show Preferences for Syntactic Formalisms?cited by 30
  10. 2019Is Multilingual BERT Fluent in Language Generation?cited by 28
  11. 2020Probing Multilingual BERT for Genetic and Typological Signalscited by 26
  12. 2020What does it mean to be language-agnostic? Probing multilingual sentence encoders for typological propertiescited by 23
  13. 2020Bridging Linguistic Typology and Multilingual Machine Translation with Multi-View Language Representationscited by 21
  14. 2020A Systematic Analysis of Morphological Content in BERT Models for Multiple Languagescited by 20
  15. 2018From Phonology to Syntax: Unsupervised Linguistic Typology at Different Levels with Language Embeddingscited by 13
  16. 2019Probing Multilingual Sentence Representations With X-Probecited by 13
  17. 2021Does Typological Blinding Impede Cross-Lingual Sharing?cited by 9

Built on by

Not cited yet.