mlsanityData to 14 Oct 2026

Harmonizing Diverse Models: A Layer-wise Merging Strategy for Consistent Generation

arXiv 2510.14915 ↗First posted 16 Oct 2025cs.CLcited by 0

On the map: Retrieval-augmented generation → RAG pipeline design and evaluation

Abstract (arXiv)

Retrieval-Augmented Generation (RAG) systems leverage Large Language Models (LLMs) to generate accurate and reliable responses that are grounded in retrieved context. However, LLMs often generate inconsistent outputs for semantically equivalent inputs, a problem compounded by the scarcity of consistency-focused training data and the limitations of current fine-tuning techniques in enhancing output consistency. We propose a new approach combining systematic synthetic data generation, triplet loss for better embeddings, and a novel layer-wise model merging approach. Using consistency-aware weights derived from intermediate layer activations, our method effectively integrates knowledge from specialized models. Experimental results how that our merged model significantly enhances output consistency, achieving a ~47.5% improvement in response similarity over the baseline, thus offering a practical solution for increasing the reliability of an industrial RAG system.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in RAG pipeline design and evaluation, one of 15 directions in Retrieval-augmented generation. Builds on 17 of its 26 references that are in the map. First posted on arXiv 16 Oct 2025.

Builds on

17 of 26 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024Retrieval-Augmented Generation for Natural Language Processing: A Survey18 Jul

Other references · 16

  1. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  2. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  3. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
  4. 2015FaceNet: A Unified Embedding for Face Recognition and Clusteringcited by 2,849
  5. 2023TIES-Merging: Resolving Interference When Merging Modelscited by 818
  6. 2023Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunchcited by 694
  7. 2021Measuring and Improving Consistency in Pretrained Language Modelscited by 364
  8. 2025Harnessing Multiple Large Language Models: A Survey on LLM Ensemblecited by 86
  9. 2022Measuring Reliability of Large Language Models through Semantic Consistencycited by 38
  10. 2024Towards Consistent Natural-Language Explanations via Explanation-Consistency Finetuningcited by 26
  11. 2024Prompt Perturbation Consistency Learning for Robust Language Modelscited by 26
  12. 2024Evaluating the Consistency of LLM Evaluatorscited by 24
4 more
  1. 2025Out of Style: RAG's Fragility to Linguistic Variationcited by 11
  2. 2025Estimating LLM Consistency: A User Baseline vs Surrogate Metricscited by 8
  3. 2025From Retrieval to Generation: Comparing Different Approachescited by 2
  4. 2024Consistency Matters: Explore LLMs Consistency From a Black-Box Perspectivecited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2023Retrieval-Augmented Generation for Large Language Models: A Survey
  2. 2026Mapping the RAG Landscape: A Four Axis Taxonomy of Efficiency, Defense, Interactivity, and Reasoning
  3. 2024A Survey on Retrieval-Augmented Text Generation for Large Language Models
  4. 2025When Retrieval Succeeds and Fails: Rethinking Retrieval-Augmented Generation for LLMs
  5. 2025Retrieval-Augmented Generation: A Comprehensive Survey of Architectures, Enhancements, and Robustness Frontiers
  6. 2024Towards Understanding Systems Trade-offs in Retrieval-Augmented Generation Model Inference
  7. 2024Deploying Large Language Models With Retrieval Augmented Generation
  8. 2025Retrieval Augmented Generation Evaluation in the Era of Large Language Models: A Comprehensive Survey