mlsanityData to 7 Oct 2026

Multi-Task Contrastive Learning for 8192-Token Bilingual Text Embeddings

arXiv 2402.17016 ↗First posted 26 Feb 2024cs.CL cs.AI cs.IRcited by 20

Not on the map. Why?

Abstract (arXiv)

We introduce a novel suite of state-of-the-art bilingual text embedding models that are designed to support English and another target language. These models are capable of processing lengthy text inputs with up to 8192 tokens, making them highly versatile for a range of natural language processing tasks such as text retrieval, clustering, and semantic textual similarity (STS) calculations. By focusing on bilingual models and introducing a unique multi-task learning objective, we have significantly improved the model performance on STS tasks, which outperforms the capabilities of existing multilingual models in both target language understanding and cross-lingual evaluation tasks. Moreover, our bilingual models are more efficient, requiring fewer parameters and less memory due to their smaller vocabulary needs. Furthermore, we have expanded the Massive Text Embedding Benchmark (MTEB) to include benchmarks for German and Spanish embedding models. This integration aims to stimulate further research and advancement in text embedding technologies for these languages.

Key facts. Cited by 20 papers in the map (data through 7 Oct 2026). Builds on 52 of its 74 references that are in the map. First posted on arXiv 26 Feb 2024.

Builds on

52 of 74 references.

References · 52

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  4. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  5. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  6. 2016Layer Normalizationcited by 5,579
  7. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  8. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
  9. 2016MS MARCO: A Human Generated MAchine Reading COmprehension Datasetcited by 1,591
  10. 2017Multi-Task Learning Using Uncertainty to Weigh Losses for Scene Geometry and Semanticscited by 1,345
  11. 2022Text Embeddings by Weakly-Supervised Contrastive Pre-trainingcited by 1,216
  12. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,199
38 more
  1. 2019Cross-lingual Language Model Pretrainingcited by 1,088
  2. 2021Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationcited by 1,085
  3. 2017SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluationcited by 998
  4. 2018XNLI: Evaluating Cross-lingual Sentence Representationscited by 913
  5. 2017GradNorm: Gradient Normalization for Adaptive Loss Balancing in Deep Multitask Networkscited by 816
  6. 2020XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalizationcited by 738
  7. 2021Deduplicating Training Data Makes Language Models Bettercited by 705
  8. 2022MTEB: Massive Text Embedding Benchmarkcited by 623
  9. 2020Making Monolingual Sentence Embeddings Multilingual using Knowledge Distillationcited by 591
  10. 2021The FLORES-101 Evaluation Benchmark for Low-Resource and Multilingual Machine Translationcited by 556
  11. 2020MAD-X: An Adapter-Based Framework for Multi-Task Cross-Lingual Transfercited by 443
  12. 2018Massively Multilingual Sentence Embeddings for Zero-Shot Cross-Lingual Transfer and Beyondcited by 434
  13. 2020Multi-Task Learning with Deep Neural Networks: A Surveycited by 374
  14. 2021XL-Sum: Large-Scale Multilingual Abstractive Summarization for 44 Languagescited by 346
  15. 2020Query-Key Normalization for Transformerscited by 266
  16. 2019PAWS-X: A Cross-lingual Adversarial Dataset for Paraphrase Identificationcited by 262
  17. 2019Massively Multilingual Neural Machine Translation in the Wild: Findings and Challengescited by 253
  18. 2022Promptagator: Few-shot Dense Retrieval From 8 Examplescited by 233
  19. 2018Sentence Encoders on STILTs: Supplementary Training on Intermediate Labeled-data Taskscited by 214
  20. 2023MADLAD-400: A Multilingual And Document-Level Large Audited Datasetcited by 180
  21. 2019Facebook FAIR's WMT19 News Translation Task Submissioncited by 179
  22. 2022Lifting the Curse of Multilinguality by Pre-training Modular Transformerscited by 145
  23. 2019Synthetic QA Corpora Generation with Roundtrip Consistencycited by 143
  24. 2020MLSUM: The Multilingual Summarization Corpuscited by 121
  25. 2020Rethinking embedding coupling in pre-trained language modelscited by 103
  26. 2023Jina Embeddings 2: 8192-Token General-Purpose Text Embeddings for Long Documentscited by 103
  27. 2015Multi-task Sequence to Sequence Learningcited by 94
  28. 2022Mintaka: A Complex, Natural, and Multilingual Dataset for End-to-End Question Answeringcited by 71
  29. 2018Learning General Purpose Distributed Sentence Representations via Large Scale Multi-task Learningcited by 52
  30. 2021GermanQuAD and GermanDPR: Improving Non-English Question Answering and Passage Retrievalcited by 39
  31. 2021MarIA: Spanish Language Modelscited by 36
  32. 2021BERT, mBERT, or BiBERT? A Study on Contextualized Embeddings for Neural Machine Translationcited by 35
  33. 2023Jina Embeddings: A Novel Set of High-Performance Sentence Embedding Modelscited by 35
  34. 2022Analyzing the Mono- and Cross-Lingual Pretraining Dynamics of Multilingual Language Modelscited by 32
  35. 2021Cross-Market Product Recommendationcited by 22
  36. 2021Allocating Large Vocabulary Capacity for Cross-lingual Language Model Pre-trainingcited by 22
  37. 2022Perplexed by Quality: A Perplexity-based Method for Adult and Harmful Content Detection in Multilingual Heterogeneous Web Datacited by 22
  38. 2020FinEst BERT and CroSloEngual BERT: less is more in multilingual modelscited by 20

50 most cited of 52 All 52 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 11; 2025: 6; 2026: 3.

20 papers.

Sort:

By area:

  1. 2024jina-embeddings-v3: Multilingual Embeddings With Task LoRAcited by 138
  2. 2024Large Concept Models: Language Modeling in a Sentence Representation Spacecited by 86
  3. 2024BadRAG: Identifying Vulnerabilities in Retrieval Augmented Generation of Large Language Modelscited by 84
  4. 2024Jina CLIP: Your CLIP Model Is Also Your Text Retrievercited by 46
  5. 2024OmniEval: An Omnidirectional and Automatic RAG Evaluation Benchmark in Financial Domaincited by 28
  6. 2024Jina-ColBERT-v2: A General-Purpose Multilingual Late Interaction Retrievercited by 25
  7. 2024Leveraging Passage Embeddings for Efficient Listwise Reranking with Large Language Modelscited by 20
  8. 2026jina-embeddings-v5-text: Task-Targeted Embedding Distillationcited by 19
  9. 2025Should We Still Pretrain Encoders with Masked Language Modeling?cited by 15
  10. 2024Swan and ArabicMTEB: Dialect-Aware, Arabic-Centric, Cross-Lingual, and Cross-Cultural Embedding Models and Benchmarkscited by 10
  11. 2025Position: Text Embeddings Should Capture Implicit Semantics, Not Just Surface Meaningcited by 10
  12. 2024RAG4ITOps: A Supervised Fine-Tunable and Comprehensive RAG Framework for IT Operations and Maintenancecited by 7
8 more
  1. 2026LMEB: Long-horizon Memory Embedding Benchmarkcited by 6
  2. 2025ConFit v2: Improving Resume-Job Matching using Hypothetical Resume Embedding and Runner-Up Hard-Negative Miningcited by 4
  3. 2025Multimodal Assessment of Classroom Discourse Quality: A Text-Centered Attention-Based Multi-Task Learning Approachcited by 2
  4. 2025Culinary Crossroads: A RAG Framework for Enhancing Diversity in Cross-Cultural Recipe Adaptationcited by 2
  5. 2026An Extreme Multi-label Text Classification (XMTC) Library Dataset: What if we took "Use of Practical AI in Digital Libraries" seriously?cited by 2
  6. 2024Optimal and efficient text counterfactuals using Graph Neural Networkscited by 1
  7. 2024On the Suitability of pre-trained foundational LLMs for Analysis in German Legal Educationcited by 1
  8. 2025Chinese ModernBERT with Whole-Word Maskingcited by 0