mlsanityData to 7 Oct 2026

Multilingual Text Representation

arXiv 2309.00949 ↗First posted 2 Sep 2023cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

Modern NLP breakthrough includes large multilingual models capable of performing tasks across more than 100 languages. State-of-the-art language models came a long way, starting from the simple one-hot representation of words capable of performing tasks like natural language understanding, common-sense reasoning, or question-answering, thus capturing both the syntax and semantics of texts. At the same time, language models are expanding beyond our known language boundary, even competitively performing over very low-resource dialects of endangered languages. However, there are still problems to solve to ensure an equitable representation of texts through a unified modeling space across language and speakers. In this survey, we shed light on this iterative progression of multilingual text representation and discuss the driving factors that ultimately led to the current state-of-the-art. Subsequently, we discuss how the full potential of language democratization could be obtained, reaching beyond the known limits and what is the scope of improvement in that space.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 51 of its 75 references that are in the map. First posted on arXiv 2 Sep 2023.

Builds on

51 of 75 references.

References · 51

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  6. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,530
  7. 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,346
  8. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
  9. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  10. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
  11. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
  12. 2018Deep contextualized word representationscited by 2,285
38 more
  1. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,250
  2. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,973
  3. 2016Enriching Word Vectors with Subword Informationcited by 1,578
  4. 2014On the Properties of Neural Machine Translation: Encoder-Decoder Approachescited by 1,303
  5. 2019Cross-lingual Language Model Pretrainingcited by 1,088
  6. 2018XNLI: Evaluating Cross-lingual Sentence Representationscited by 913
  7. 2020The State and Fate of Linguistic Diversity and Inclusion in the NLP Worldcited by 789
  8. 2020XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalizationcited by 738
  9. 2018Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidatescited by 632
  10. 2020MAD-X: An Adapter-Based Framework for Multi-Task Cross-Lingual Transfercited by 443
  11. 2018Massively Multilingual Sentence Embeddings for Zero-Shot Cross-Lingual Transfer and Beyondcited by 434
  12. 2016Deep Biaffine Attention for Neural Dependency Parsingcited by 334
  13. 2020How Good is Your Tokenizer? On the Monolingual Performance of Multilingual Language Modelscited by 313
  14. 2021Towards Understanding and Mitigating Social Biases in Language Modelscited by 294
  15. 2020The Lottery Ticket Hypothesis for Pre-trained BERT Networkscited by 275
  16. 2021CANINE: Pre-training an Efficient Tokenization-Free Encoder for Language Representationcited by 229
  17. 2013Exploiting Similarities among Languages for Machine Translationcited by 187
  18. 2021Word Alignment by Fine-tuning Embeddings on Parallel Corporacited by 186
  19. 2021Masked Language Modeling and the Distributional Hypothesis: Order Word Matters Pre-training for Littlecited by 176
  20. 2021XTREME-R: Towards More Challenging and Nuanced Multilingual Evaluationcited by 167
  21. 2021Charformer: Fast Character Transformers via Gradient-based Subword Tokenizationcited by 158
  22. 2022Lifting the Curse of Multilinguality by Pre-training Modular Transformerscited by 145
  23. 2022mGPT: Few-Shot Learners Go Multilingualcited by 141
  24. 2020When Being Unseen from mBERT is just the Beginning: Handling New Languages With Multilingual Language Modelscited by 131
  25. 2021Between words and characters: A Brief History of Open-Vocabulary Modeling and Tokenization in NLPcited by 130
  26. 2020On the Effect of Dropping Layers of Pre-trained Transformer Modelscited by 118
  27. 2021Systematic Inequalities in Language Technology Performance across the World's Languagescited by 118
  28. 2020When BERT Plays the Lottery, All Tickets Are Winningcited by 114
  29. 2020Multilingual Alignment of Contextual Word Representationscited by 113
  30. 2017Learned in Translation: Contextualized Word Vectorscited by 104
  31. 2020Extending Multilingual BERT to Low-Resource Languagescited by 88
  32. 2021Mitigating Language-Dependent Ethnic Bias in BERTcited by 87
  33. 2022Language Modelling with Pixelscited by 53
  34. 2022Discovering Language-neutral Sub-networks in Multilingual Language Modelscited by 28
  35. 2022Using Natural Sentences for Understanding Biases in Language Modelscited by 19
  36. 2022Same Neurons, Different Languages: Probing Morphosyntax in Multilingual Pre-trained Modelscited by 14
  37. 2016Character-Level Language Modeling with Hierarchical Recurrent Neural Networkscited by 13
  38. 2021Probing as Quantifying Inductive Biascited by 11

50 most cited of 51 All 51 (JSON)

Built on by

Not cited yet.