mlsanityData to 13 Oct 2026

Bootstrapping Embeddings for Low Resource Languages

arXiv 2603.01732 ↗First posted 2 Mar 2026cs.CLcited by 0

On the map: Synthetic data and self-improvement for LLMs → LLM synthetic data generation

Abstract (arXiv)

Embedding models are crucial to modern NLP. However, the creation of the most effective models relies on carefully constructed supervised finetuning data. For high resource languages, such as English, such datasets are readily available. However, for hundreds of other languages, they are simply non-existent. We investigate whether the advent of large language models can help to bridge this gap. We test three different strategies for generating synthetic triplet data used to optimise embedding models. These include in-context learning as well as two novel approaches, leveraging adapter composition and cross lingual finetuning of the LLM generator (XL-LoRA) respectively. We find that while in-context learning still falls short of strong non-synthetic baselines, adapter composition and XL-LoRA yield strong performance gains across a wide array of tasks and languages, offering a clear, scalable pathway to producing performant embedding models for a wide variety of languages.

Key facts. No citing papers in the map yet (data through 13 Oct 2026). Placed in LLM synthetic data generation, one of 13 directions in Synthetic data and self-improvement for LLMs. Builds on 35 of its 51 references that are in the map. First posted on arXiv 2 Mar 2026.

Builds on

35 of 51 references.

References · 35

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,304
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,231
  3. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,009
  4. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,221
  5. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,107
  6. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,916
  7. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,693
  8. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,676
  9. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,586
  10. 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,170
  11. 2015A large annotated corpus for learning natural language inferencecited by 1,841
  12. 2020Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hyperspherecited by 1,397
23 more
  1. 2021BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Modelscited by 1,237
  2. 2018XNLI: Evaluating Cross-lingual Sentence Representationscited by 914
  3. 2020Language-agnostic BERT Sentence Embeddingcited by 622
  4. 2020Making Monolingual Sentence Embeddings Multilingual using Knowledge Distillationcited by 591
  5. 2024Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder for Fast, Memory Efficient, and Long Context Finetuning and Inferencecited by 544
  6. 2018Universal Sentence Encodercited by 488
  7. 2021Sentence-T5: Scalable Sentence Encoders from Pre-trained Text-to-Text Modelscited by 462
  8. 2022ToxiGen: A Large-Scale Machine-Generated Dataset for Adversarial and Implicit Hate Speech Detectioncited by 421
  9. 2022Unnatural Instructions: Tuning Language Models with (Almost) No Human Laborcited by 395
  10. 2024Do Llamas Work in English? On the Latent Language of Multilingual Transformerscited by 280
  11. 2017Unsupervised Machine Translation Using Monolingual Corpora Onlycited by 272
  12. 2021Whitening Sentence Representations for Better Semantics and Faster Retrievalcited by 200
  13. 2023Composing Parameter-Efficient Modules with Arithmetic Operationscited by 155
  14. 2022Transfer Learning Approaches for Building Cross-Language Dense Retrieval Modelscited by 36
  15. 2024AdaMergeX: Cross-Lingual Transfer with Large Language Models via Adaptive Adapter Mergingcited by 34
  16. 2022PCL: Peer-Contrastive Learning with Diverse Augmentations for Unsupervised Sentence Embeddingscited by 29
  17. 2023Contrastive Learning of Sentence Embeddings from Scratchcited by 24
  18. 2021Paraphrastic Representations at Scalecited by 21
  19. 2023Low-Rank Adaptation for Multilingual Summarization: An Empirical Studycited by 13
  20. 2023LEALLA: Learning Lightweight Language-agnostic Sentence Embeddings with Knowledge Distillationcited by 11
  21. 2025The AI Language Proficiency Monitor -- Tracking the Progress of LLMs on Multilingual Benchmarkscited by 7
  22. 2024Self-StrAE at SemEval-2024 Task 1: Making Self-Structuring AutoEncoders Learn More With Lesscited by 3
  23. 2024Banyan: Improved Representation Learning with Explicit Structurecited by 3

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Better as Generators Than Classifiers: Leveraging LLMs and Synthetic Data for Low-Resource Multilingual Classification
  2. 2026Less is More: Adapting Text Embeddings for Low-Resource Languages with Small Scale Noisy Synthetic Data
  3. 2025Towards Inclusive NLP: Assessing Compressed Multilingual Transformers across Diverse Language Benchmarks
  4. 2025Franken-Adapter: Cross-Lingual Adaptation of LLMs by Embedding Surgery
  5. 2022Bootstrapping Multilingual Semantic Parsers using Large Language Models
  6. 2023ToolkenGPT: Augmenting Frozen Language Models with Massive Tools via Tool Embeddings
  7. 2024How Can We Effectively Expand the Vocabulary of LLMs with 0.01GB of Target Language Text?
  8. 2023Mean BERTs make erratic language teachers: the effectiveness of latent bootstrapping in low-resource settings