One Instruction Does Not Fit All: How Well Do Embeddings Align Personas and Instructions in Low-Resource Indian Languages?
Not on the map. Why?
Abstract (arXiv)
Aligning multilingual assistants with culturally grounded user preferences is essential for serving India's linguistically diverse population of over one billion speakers across multiple scripts. However, existing benchmarks either focus on a single language or conflate retrieval with generation, leaving open the question of whether current embedding models can encode persona-instruction compatibility without relying on response synthesis. We present a unified benchmark spanning 12 Indian languages and four evaluation tasks: monolingual and cross-lingual persona-to-instruction retrieval, reverse retrieval from instruction to persona, and binary compatibility classification. Eight multilingual embedding models are evaluated in a frozen-encoder setting with a thin logistic regression head for classification. E5-Large-Instruct achieves the highest Recall@1 of 27.4% on monolingual retrieval and 20.7% on cross-lingual transfer, while BGE-M3 leads reverse retrieval at 32.1% Recall@1. For classification, LaBSE attains 75.3% AUROC with strong calibration. These findings offer practical guidance for model selection in Indic multilingual retrieval and establish reproducible baselines for future workfootnoteCode, datasets, and models are publicly available at https://github.com/aryashah2k/PI-Indic-Align.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 32 of its 49 references that are in the map. First posted on arXiv 15 Jan 2026.
Builds on
32 of 49 references.
References · 32
- 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
- 2017On Calibration of Modern Neural Networkscited by 4,860
- 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
- 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,168
- 2016MS MARCO: A Human Generated MAchine Reading COmprehension Datasetcited by 1,591
- 2021BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Modelscited by 1,225
- 2021Unsupervised Dense Information Retrieval with Contrastive Learningcited by 1,203
- 2024M3-Embedding: Multi-Linguality, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillationcited by 1,050
- 2018Personalizing Dialogue Agents: I have a dog, do you have pets too?cited by 820
- 2020XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalizationcited by 738
- 2020TyDi QA: A Benchmark for Information-Seeking Question Answering in Typologically Diverse Languagescited by 486
- 2021Sentence-T5: Scalable Sentence Encoders from Pre-trained Text-to-Text Modelscited by 462
20 more
- 2024Scaling Synthetic Data Creation with 1,000,000,000 Personascited by 397
- 2024Multilingual E5 Text Embeddings: A Technical Reportcited by 357
- 2022One Embedder, Any Task: Instruction-Finetuned Text Embeddingscited by 329
- 2020Calibration of Pre-trained Transformerscited by 307
- 2024mGTE: Generalized Long-Context Text Representation and Reranking Models for Multilingual Text Retrievalcited by 233
- 2024Generative Representational Instruction Tuningcited by 227
- 2021MuRIL: Multilingual Representations for Indian Languagescited by 177
- 2021XTREME-R: Towards More Challenging and Nuanced Multilingual Evaluationcited by 167
- 2023IndicTrans2: Towards High-Quality and Accessible Machine Translation Models for all 22 Scheduled Indian Languagescited by 149
- 2018Training Millions of Personalized Dialogue Agentscited by 126
- 2021Mr. TyDi: A Multi-lingual Benchmark for Dense Retrievalcited by 118
- 2021mMARCO: A Multilingual Version of the MS MARCO Passage Ranking Datasetcited by 116
- 2022Towards Leaving No Indic Language Behind: Building Monolingual Corpora, Benchmark and Models for Indic Languagescited by 99
- 2024FollowIR: Evaluating and Teaching Information Retrieval Models to Follow Instructionscited by 67
- 2022Making a MIRACL: Multilingual Information Retrieval Across a Continuum of Languagescited by 47
- 2020LAReQA: Language-agnostic answer retrieval from a multilingual poolcited by 41
- 2022L3Cube-HindBERT and DevBERT: Pre-Trained BERT Transformer models for Devanagari based Hindi and Marathi Languagescited by 35
- 2024Answer is All You Need: Instruction-following Text Embedding via Answering the Questioncited by 26
- 2019Dually Interactive Matching Network for Personalized Response Selection in Retrieval-Based Chatbotscited by 23
- 2024Learning Retrieval Augmentation for Personalized Dialogue Generationcited by 18
Built on by
Not cited yet.