mlsanityData to 14 Oct 2026

Improving Cross-lingual Representation for Semantic Retrieval with Code-switching

arXiv 2403.01364 ↗First posted 3 Mar 2024cs.CLcited by 1

Not on the map. Why?

Abstract (arXiv)

Semantic Retrieval (SR) has become an indispensable part of the FAQ system in the task-oriented question-answering (QA) dialogue scenario. The demands for a cross-lingual smart-customer-service system for an e-commerce platform or some particular business conditions have been increasing recently. Most previous studies exploit cross-lingual pre-trained models (PTMs) for multi-lingual knowledge retrieval directly, while some others also leverage the continual pre-training before fine-tuning PTMs on the downstream tasks. However, no matter which schema is used, the previous work ignores to inform PTMs of some features of the downstream task, i.e. train their PTMs without providing any signals related to SR. To this end, in this work, we propose an Alternative Cross-lingual PTM for SR via code-switching. We are the first to utilize the code-switching approach for cross-lingual SR. Besides, we introduce the novel code-switched continual pre-training instead of directly using the PTMs on the SR tasks. The experimental results show that our proposed approach consistently outperforms the previous SOTA methods on SR and semantic textual similarity (STS) tasks with three business corpora and four open datasets in 20+ languages.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 44 of its 69 references that are in the map. First posted on arXiv 3 Mar 2024.

Builds on

44 of 69 references.

References · 44

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  5. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  6. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,935
  7. 2017Billion-scale similarity search with GPUscited by 2,497
  8. 2018Deep contextualized word representationscited by 2,290
  9. 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,173
  10. 2019Cross-lingual Language Model Pretrainingcited by 1,090
  11. 2016ConceptNet 5.5: An Open Multilingual Graph of General Knowledgecited by 1,067
  12. 2020Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrievalcited by 849
32 more
  1. 2020XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalizationcited by 742
  2. 2020Language-agnostic BERT Sentence Embeddingcited by 627
  3. 2020Making Monolingual Sentence Embeddings Multilingual using Knowledge Distillationcited by 593
  4. 2019CCNet: Extracting High Quality Monolingual Datasets from Web Crawl Datacited by 548
  5. 2017Word Translation Without Parallel Datacited by 487
  6. 2018Massively Multilingual Sentence Embeddings for Zero-Shot Cross-Lingual Transfer and Beyondcited by 434
  7. 2017Supervised Learning of Universal Sentence Representations from Natural Language Inference Datacited by 432
  8. 2015Skip-Thought Vectorscited by 322
  9. 2020InfoXLM: An Information-Theoretic Framework for Cross-Lingual Language Model Pre-Trainingcited by 237
  10. 2019Multilingual Universal Sentence Encoder for Semantic Retrievalcited by 193
  11. 2019Unicoder: A Universal Language Encoder by Pre-training with Multiple Cross-lingual Taskscited by 96
  12. 2016Learning Distributed Representations of Sentences from Unlabelled Datacited by 91
  13. 2020CoSDA-ML: Multi-Lingual Code-Switching Data Augmentation for Zero-Shot Cross-Lingual NLPcited by 79
  14. 2020Pre-training Multilingual Neural Machine Translation by Leveraging Alignment Informationcited by 64
  15. 2019Cross-Lingual Natural Language Generation via Pre-Trainingcited by 63
  16. 2020ERNIE-M: Enhanced Multilingual Representation by Aligning Cross-lingual Semantics with Monolingual Corporacited by 63
  17. 2020SemEval-2020 Task 9: Overview of Sentiment Analysis of Code-Mixed Tweetscited by 62
  18. 2019Improving Multilingual Sentence Embedding using Bi-directional Dual Encoder with Additive Margin Softmaxcited by 58
  19. 2020On Learning Universal Representations Across Languagescited by 58
  20. 2020Cross-lingual Retrieval for Iterative Self-Supervised Trainingcited by 54
  21. 2020Code-switching pre-training for neural machine translationcited by 43
  22. 2020VECO: Variable and Flexible Cross-lingual Pre-training for Language Understanding and Generationcited by 34
  23. 2020Universal Sentence Representation Learning with Conditional Masked Language Modelcited by 34
  24. 2020A New Dataset for Natural Language Inference from Code-mixed Conversationscited by 29
  25. 2015Semi-supervised Question Retrieval with Gated Convolutionscited by 28
  26. 2019Evaluating the Cross-Lingual Effectiveness of Massively Multilingual Neural Machine Translationcited by 27
  27. 2021Multilingual Code-Switching for Zero-Shot Cross-Lingual Intent Prediction and Slot Fillingcited by 25
  28. 2018Universal Dependency Parsing for Hindi-English Code-switchingcited by 18
  29. 2023Improving Cross-lingual Information Retrieval on Low-Resource Languages via Optimal Transport Distillationcited by 15
  30. 2023CL2CM: Improving Cross-Lingual Cross-Modal Retrieval via Cross-Lingual Knowledge Transfercited by 10
  31. 2020kk2018 at SemEval-2020 Task 9: Adversarial Training for Code-Mixing Sentiment Classificationcited by 7
  32. 2022Auto-MLM: Improved Contrastive Learning for Self-supervised Multi-lingual Knowledge Retrievalcited by 1

Built on by

Cited by year

Citing papers per year: 2025: 1.

1 paper.

  1. 2025Beyond Monolingual Assumptions: A Survey of Code-Switched NLP in the Era of Large Language Models across Modalitiescited by 5