mlsanityData to 7 Oct 2026

Modular Expert Merging for Biomedical Retrieval

arXiv 2602.04731 ↗First posted 4 Feb 2026cs.CL cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Adapting general-purpose LLMs into domain-specialized dense retrievers typically requires large-scale training on mixed-domain data. We show that merging independently trained domain-specialized experts consistently exceeds this approach across four decoder-only LLM families (0.6B-7B), four merging methods, and twelve medical and general retrieval tasks from MTEB, suggesting that parameter-space composition captures complementary domain strengths that large-scale mixed-domain training averages out. To further maximize expert quality, we introduce Synthesize-Train-Merge (STM), a modular framework that synthesizes hard negatives with a top-tier LLM and fine-tunes domain-specialized experts via LoRA before merging them, without continual pre-training. Synthesized hard negatives yield the largest gains for smaller models, and STM achieves strong performance on biomedical retrieval tasks while maintaining competitive general-domain results across all four backbone families.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 47 of its 69 references that are in the map. First posted on arXiv 4 Feb 2026.

Builds on

47 of 69 references.

References · 47

  1. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  2. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  3. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  4. 2020Scaling Laws for Neural Language Modelscited by 6,411
  5. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,779
  6. 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,168
  7. 2015A large annotated corpus for learning natural language inferencecited by 1,837
  8. 2016MS MARCO: A Human Generated MAchine Reading COmprehension Datasetcited by 1,591
  9. 2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,366
  10. 2018FEVER: a large-scale dataset for Fact Extraction and VERificationcited by 1,362
  11. 2021BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Modelscited by 1,225
  12. 2021Unsupervised Dense Information Retrieval with Contrastive Learningcited by 1,203
35 more
  1. 2025Qwen3 Embedding: Advancing Text Embedding and Reranking Through Foundation Modelscited by 1,175
  2. 2020Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrievalcited by 846
  3. 2023TIES-Merging: Resolving Interference When Merging Modelscited by 795
  4. 2023Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunchcited by 679
  5. 2019Linear Mode Connectivity and the Lottery Ticket Hypothesiscited by 627
  6. 2019ELI5: Long Form Question Answeringcited by 567
  7. 2020Fact or Fiction: Verifying Scientific Claimscited by 546
  8. 2024NV-Embed: Improved Techniques for Training LLMs as Generalist Embedding Modelscited by 466
  9. 2021Large Dual Encoders Are Generalizable Retrieverscited by 454
  10. 2020SPECTER: Document-level Representation Learning using Citation-informed Transformerscited by 426
  11. 2024LLM2Vec: Large Language Models Are Secretly Powerful Text Encoderscited by 396
  12. 2024Benchmarking Retrieval-Augmented Generation for Medicinecited by 326
  13. 2023ChatDoctor: A Medical Chat Model Fine-Tuned on a Large Language Model Meta-AI (LLaMA) Using Medical Domain Knowledgecited by 324
  14. 2024BioMistral: A Collection of Open-Source Pretrained Large Language Models for Medical Domainscited by 299
  15. 2024Evolutionary Optimization of Model Merging Recipescited by 204
  16. 2024Arcee's MergeKit: A Toolkit for Merging Large Language Modelscited by 189
  17. 2021Optimizing Dense Retrieval Model Training with Hard Negativescited by 154
  18. 2020Linear Mode Connectivity in Multitask and Continual Learningcited by 144
  19. 2019A Question-Entailment Approach to Question Answeringcited by 133
  20. 2025EmbeddingGemma: Powerful and Lightweight Text Representationscited by 128
  21. 2024Reducing hallucination in structured outputs via Retrieval-Augmented Generationcited by 92
  22. 2024BM25S: Orders of magnitude faster lexical search via eager sparse scoringcited by 87
  23. 2024Blended RAG: Improving RAG (Retriever-Augmented Generation) Accuracy with Semantic Search and Hybrid Query-Based Retrieverscited by 83
  24. 2024NV-Retriever: Improving text embedding models with effective hard-negative miningcited by 83
  25. 2020Effective Transfer Learning for Identifying Similar Questions: Matching User Questions to COVID-19 FAQscited by 48
  26. 2025Llama-Embed-Nemotron-8B: A Universal Text Embedding Model for Multilingual and Cross-Lingual Taskscited by 46
  27. 2023LM-Cocktail: Resilient Tuning of Language Models via Model Mergingcited by 44
  28. 2024Promptriever: Instruction-Trained Retrievers Can Be Prompted Like Language Modelscited by 39
  29. 2024BMRetriever: Tuning Large Language Models as Better Biomedical Text Retrieverscited by 38
  30. 2022Using Interactive Feedback to Improve the Accuracy and Explainability of Question Answering Systems Post-Deploymentcited by 25
  31. 2025A Modular Approach for Clinical SLMs Driven by Synthetic Data with Pre-Instruction Tuning, Model Merging, and Clinical-Tasks Alignmentcited by 20
  32. 2021Searching for Scientific Evidence in a Pandemic: An Overview of TREC-COVIDcited by 19
  33. 2024SyNeg: LLM-Driven Synthetic Hard-Negatives for Dense Retrievalcited by 14
  34. 2024Scalable Data Ablation Approximations for Language Models through Modular Training and Mergingcited by 11
  35. 2024CURE: A Dataset for Clinical Understanding & Retrieval Evaluationcited by 5

Built on by

Not cited yet.