mlsanityData to 14 Oct 2026

Generalistic or Specific Embeddings, Which is Better? An Empirical Study on Search for Clinical Coding in Non-English Languages

arXiv 2605.30529 ↗First posted 28 May 2026cs.CL cs.AI cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Sentence-embedding models for semantic search are overwhelmingly developed and evaluated on English corpora. When applied to clinical retrieval in other languages -- particularly retrieval of ICD-10-CM / CIE-10 codes -- recall degrades in ways often masked by aggregate benchmarks. We study whether large generative language models can serve as data factories to close this gap. We build a two-stage retriever (bi-encoder followed by cross-encoder reranker), fine-tuned from a Spanish biomedical encoder (PlanTL-GOB-ES/bsc-bio-ehr-es) on Gemini-generated synthetic data covering English, Spanish, Catalan, Italian, Portuguese and French, and evaluate against BioBERT-ST and the un-tuned Spanish encoder. The bi-encoder alone matches BioBERT-ST on MRR (0.876 vs. 0.866) and overtakes it on R@3 (0.650 vs. 0.626) and R@5 (0.804 vs. 0.790) without English biomedical pretraining. Adding a cross-encoder reranker lifts aggregate R@5 to 0.822 and dominates on four of five languages (+0.017 Spanish, +0.033 Catalan, +0.018 French, +0.037 Portuguese) at the cost of a small English regression. The trade-off is clinically acceptable: Portuguese reaches R@5 = 0.829 vs. BioBERT-ST's 0.714. Contributions: an open recipe for building domain-specific medical retrievers from LLM-generated data; quantification of the learning gain (MRR 0.755 to 0.876, +15.9% with ~19,500 synthetic pairs); and a characterisation of where gains concentrate by language and rank.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 75 of its 100 references that are in the map. First posted on arXiv 28 May 2026.

Builds on

75 of 100 references.

References · 75

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  3. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  4. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
  5. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  6. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  7. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,182
  8. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,935
  9. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,824
  10. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,724
  11. 2019BioBERT: a pre-trained biomedical language representation model for biomedical text miningcited by 1,947
  12. 2022Large Language Models Encode Clinical Knowledgecited by 1,830
38 more
  1. 2020What Disease does this Patient Have? A Large-scale Open Domain Question Answering Dataset from Medical Examscited by 1,356
  2. 2019PubMedQA: A Dataset for Biomedical Research Question Answeringcited by 1,288
  3. 2021BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Modelscited by 1,252
  4. 2020ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERTcited by 1,178
  5. 2023Large Language Models are not Fair Evaluatorscited by 1,085
  6. 2022Large Language Models Are Human-Level Prompt Engineerscited by 1,045
  7. 2019Publicly Available Clinical BERT Embeddingscited by 1,002
  8. 2020MPNet: Masked and Permuted Pre-training for Language Understandingcited by 981
  9. 2024Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorscited by 897
  10. 2023DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelinescited by 832
  11. 2020The State and Fate of Linguistic Diversity and Inclusion in the NLP Worldcited by 798
  12. 2023ChatEval: Towards Better LLM-based Evaluators through Multi-Agent Debatecited by 742
  13. 2024LLM Evaluators Recognize and Favor Their Own Generationscited by 686
  14. 2019Passage Re-ranking with BERTcited by 645
  15. 2019How multilingual is Multilingual BERT?cited by 645
  16. 2023Enabling Large Language Models to Generate Text with Citationscited by 595
  17. 2020Making Monolingual Sentence Embeddings Multilingual using Knowledge Distillationcited by 593
  18. 2022MedMCQA : A Large-scale Multi-Subject Multi-Choice Dataset for Medical domain Question Answeringcited by 543
  19. 2024Mixture-of-Agents Enhances Large Language Model Capabilitiescited by 438
  20. 2025HealthBench: Evaluating Large Language Models Towards Improved Human Healthcited by 429
  21. 2023The Curse of Recursion: Training on Generated Data Makes Models Forgetcited by 356
  22. 2023NLP Evaluation in trouble: On the Need to Measure LLM Data Contamination for each Benchmarkcited by 348
  23. 2024Benchmarking Retrieval-Augmented Generation for Medicinecited by 331
  24. 2020How Good is Your Tokenizer? On the Monolingual Performance of Multilingual Language Modelscited by 318
  25. 2023MedAgents: Large Language Models as Collaborators for Zero-shot Medical Reasoningcited by 316
  26. 2019Beto, Bentz, Becas: The Surprising Cross-Lingual Effectiveness of BERTcited by 301
  27. 2022Teaching language models to support answers with verified quotescited by 292
  28. 2024On LLMs-Driven Synthetic Data Generation, Curation, and Evaluation: A Surveycited by 244
  29. 2020Self-Alignment Pretraining for Biomedical Entity Representationscited by 237
  30. 2022Promptagator: Few-shot Dense Retrieval From 8 Examplescited by 235
  31. 2018Explainable Prediction of Medical Codes from Clinical Textcited by 234
  32. 2024AgentClinic: a multimodal agent benchmark to evaluate AI in simulated clinical environmentscited by 231
  33. 2017Efficient Natural Language Response Suggestion for Smart Replycited by 229
  34. 2023Language Model Tokenizers Introduce Unfairness Between Languagescited by 214
  35. 2022Can large language models reason about medical questions?cited by 194
  36. 2023Do All Languages Cost the Same? Tokenization in the Era of Commercial Language Modelscited by 192
  37. 2024Lessons from the Trenches on Reproducible Evaluation of Language Modelscited by 179
  38. 2023A Study of Generative Large Language Model for Medical Research and Healthcarecited by 163

50 most cited of 75 All 75 (JSON)

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025MedEIR: A Specialized Medical Embedding Model for Enhanced Information Retrieval
  2. 2024Lessons Learned on Information Retrieval in Electronic Health Records: A Comparison of Embedding Models and Pooling Strategies
  3. 2026Benchmarking Retrieval Strategies for Biomedical Retrieval-Augmented Generation: A Controlled Empirical Study
  4. 2026MMed-Bench-IR: A Heterogeneous Benchmark for Multilingual Medical Information Retrieval
  5. 2024Development and Testing of Retrieval Augmented Generation in Large Language Models -- A Case Study Report
  6. 2024Benchmarking Retrieval-Augmented Large Language Models in Biomedical NLP: Application, Robustness, and Self-Awareness
  7. 2025Leveraging Language Models and RAG for Efficient Knowledge Discovery in Clinical Environments
  8. 2026ctELM: Decoding and Manipulating Embeddings of Clinical Trials with Embedding Language Models