# Harmonizing Diverse Models: A Layer-wise Merging Strategy for Consistent Generation

arXiv: https://arxiv.org/abs/2510.14915 · HTML: https://mlsanity.com/paper/2510.14915/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in RAG pipeline design and evaluation, one of 15 directions in Retrieval-augmented generation.
- Builds on 17 of its 26 references that are in the map.
- First posted on arXiv 16 Oct 2025.

## Placed in

- Retrieval-augmented generation / [RAG pipeline design and evaluation](https://mlsanity.com/rag/rag-pipeline-design-and-evaluation/)

## Builds on (most cited first, 17 of 17)

- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Self-Consistency Improves Chain of Thought Reasoning in Language Models](https://mlsanity.com/paper/2203.11171/) (2022)
- [FaceNet: A Unified Embedding for Face Recognition and Clustering](https://mlsanity.com/paper/1503.03832/) (2015)
- [TIES-Merging: Resolving Interference When Merging Models](https://mlsanity.com/paper/2306.01708/) (2023)
- [Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunch](https://mlsanity.com/paper/2311.03099/) (2023)
- [Measuring and Improving Consistency in Pretrained Language Models](https://mlsanity.com/paper/2102.01017/) (2021)
- [Harnessing Multiple Large Language Models: A Survey on LLM Ensemble](https://mlsanity.com/paper/2502.18036/) (2025)
- [Retrieval-Augmented Generation for Natural Language Processing: A Survey](https://mlsanity.com/paper/2407.13193/) (2024)
- [Measuring Reliability of Large Language Models through Semantic Consistency](https://mlsanity.com/paper/2211.05853/) (2022)
- [Towards Consistent Natural-Language Explanations via Explanation-Consistency Finetuning](https://mlsanity.com/paper/2401.13986/) (2024)
- [Prompt Perturbation Consistency Learning for Robust Language Models](https://mlsanity.com/paper/2402.15833/) (2024)
- [Evaluating the Consistency of LLM Evaluators](https://mlsanity.com/paper/2412.00543/) (2024)
- [Out of Style: RAG's Fragility to Linguistic Variation](https://mlsanity.com/paper/2504.08231/) (2025)
- [Estimating LLM Consistency: A User Baseline vs Surrogate Metrics](https://mlsanity.com/paper/2505.23799/) (2025)
- [From Retrieval to Generation: Comparing Different Approaches](https://mlsanity.com/paper/2502.20245/) (2025)
- [Consistency Matters: Explore LLMs Consistency From a Black-Box Perspective](https://mlsanity.com/paper/2402.17411/) (2024)

## Abstract (arXiv)

Retrieval-Augmented Generation (RAG) systems leverage Large Language Models (LLMs) to generate accurate and reliable responses that are grounded in retrieved context. However, LLMs often generate inconsistent outputs for semantically equivalent inputs, a problem compounded by the scarcity of consistency-focused training data and the limitations of current fine-tuning techniques in enhancing output consistency. We propose a new approach combining systematic synthetic data generation, triplet loss for better embeddings, and a novel layer-wise model merging approach. Using consistency-aware weights derived from intermediate layer activations, our method effectively integrates knowledge from specialized models. Experimental results how that our merged model significantly enhances output consistency, achieving a ~47.5% improvement in response similarity over the baseline, thus offering a practical solution for increasing the reliability of an industrial RAG system.