mlsanityData to 7 Oct 2026

Multilingual Sentence-T5: Scalable Sentence Encoders for Multilingual Applications

arXiv 2403.17528 ↗First posted 26 Mar 2024cs.CLcited by 7

Not on the map. Why?

Abstract (arXiv)

Prior work on multilingual sentence embedding has demonstrated that the efficient use of natural language inference (NLI) data to build high-performance models can outperform conventional methods. However, the potential benefits from the recent ``exponential'' growth of language models with billions of parameters have not yet been fully explored. In this paper, we introduce Multilingual Sentence T5 (m-ST5), as a larger model of NLI-based multilingual sentence embedding, by extending Sentence T5, an existing monolingual model. By employing the low-rank adaptation (LoRA) technique, we have achieved a successful scaling of the model's size to 5.7 billion parameters. We conducted experiments to evaluate the performance of sentence embedding and verified that the method outperforms the NLI-based prior approach. Furthermore, we also have confirmed a positive correlation between the size of the model and its performance. It was particularly noteworthy that languages with fewer resources or those with less linguistic similarity to English benefited more from the parameter increase. Our model is available at https://huggingface.co/pkshatech/m-ST5.

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Builds on 34 of its 45 references that are in the map. First posted on arXiv 26 Mar 2024.

Builds on

34 of 45 references.

References · 34

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  4. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  5. 2020Scaling Laws for Neural Language Modelscited by 6,411
  6. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
  7. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
  8. 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,168
  9. 2015A large annotated corpus for learning natural language inferencecited by 1,837
  10. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,813
  11. 2019Cross-lingual Language Model Pretrainingcited by 1,088
  12. 2017SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluationcited by 998
22 more
  1. 2018XNLI: Evaluating Cross-lingual Sentence Representationscited by 913
  2. 2020XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalizationcited by 738
  3. 2019How multilingual is Multilingual BERT?cited by 642
  4. 2020Making Monolingual Sentence Embeddings Multilingual using Knowledge Distillationcited by 591
  5. 2021Sentence-T5: Scalable Sentence Encoders from Pre-trained Text-to-Text Modelscited by 462
  6. 2018Massively Multilingual Sentence Embeddings for Zero-Shot Cross-Lingual Transfer and Beyondcited by 434
  7. 2015Skip-Thought Vectorscited by 321
  8. 2020CLUE: A Chinese Language Understanding Evaluation Benchmarkcited by 277
  9. 2020Revisiting Pre-Trained Models for Chinese Natural Language Processingcited by 262
  10. 2021ConSERT: A Contrastive Framework for Self-Supervised Sentence Representation Transfercited by 231
  11. 2022SGPT: GPT Sentence Embeddings for Semantic Searchcited by 204
  12. 2018An efficient framework for learning sentence representationscited by 197
  13. 2019Multilingual Universal Sentence Encoder for Semantic Retrievalcited by 192
  14. 2021COCO-LM: Correcting and Contrasting Text Sequences for Language Model Pretrainingcited by 126
  15. 2020Multilingual Alignment of Contextual Word Representationscited by 113
  16. 2021KLUE: Korean Language Understanding Evaluationcited by 113
  17. 2021Self-Guided Contrastive Learning for BERT Sentence Representationscited by 103
  18. 2021XLM-E: Cross-lingual Language Model Pre-training via ELECTRAcited by 72
  19. 2020On Learning Universal Representations Across Languagescited by 58
  20. 2020KorNLI and KorSTS: New Benchmark Datasets for Korean Natural Language Understandingcited by 37
  21. 2022English Contrastive Learning Can Learn Universal Cross-lingual Sentence Embeddingscited by 26
  22. 2019Adversarial Learning with Contextual Embeddings for Zero-resource Cross-lingual Classification and NERcited by 25

Built on by

Cited by year

Citing papers per year: 2024: 2; 2025: 5.

7 papers.

  1. 2025Redundancy, Isotropy, and Intrinsic Dimensionality of Prompt-based Text Embeddingscited by 10
  2. 2024JMedBench: A Benchmark for Evaluating Japanese Biomedical Large Language Modelscited by 8
  3. 2025On The Role of Pretrained Language Models in General-Purpose Text Embeddings: A Surveycited by 3
  4. 2024Modular Sentence Encoders: Separating Language Specialization from Cross-Lingual Alignmentcited by 2
  5. 2025Exploring Pose-based Sign Language Translation: Ablation Studies and Attention Insightscited by 2
  6. 2025Saudi Sign Language Translation Using T5cited by 1
  7. 2025UniBERT: Adversarial Training for Language-Universal Representationscited by 0