mlsanityData to 7 Oct 2026

ORBIT: Preserving Foundational Language Capabilities in GenRetrieval via Origin-Regulated Merging

arXiv 2605.12419 ↗First posted 12 May 2026cs.CL cs.IR cs.LGcited by 1

On the map: LLM reasoning → Reasoning benchmarks and evaluation

Abstract (arXiv)

Despite the rapid advancements in large language model (LLM) development, fine-tuning them for specific tasks often results in the catastrophic forgetting of their general, language-based reasoning abilities. This work investigates and addresses this challenge in the context of the Generative Retrieval (GenRetrieval) task. During GenRetrieval fine-tuning, we find this forgetting occurs rapidly and correlates with the distance between the fine-tuned and original model parameters. Given these observations, we propose ORBIT, a novel approach that actively tracks the distance between fine-tuned and initial model weights, and uses a weight averaging strategy to constrain model drift during GenRetrieval fine-tuning when this inter-model distance exceeds a maximum threshold. Our results show that ORBIT retains substantial text and retrieval performance by outperforming both common continual learning baselines and related regularization methods that also employ weight averaging.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 24 of its 25 references that are in the map. First posted on arXiv 12 May 2026.

Builds on

24 of 25 references.

References · 24

  1. 2016Overcoming catastrophic forgetting in neural networkscited by 4,718
  2. 2025Gemma 3 Technical Reportcited by 1,686
  3. 2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,366
  4. 2023TIES-Merging: Resolving Interference When Merging Modelscited by 795
  5. 2021Robust fine-tuning of zero-shot modelscited by 785
  6. 2021SoundStream: An End-to-End Neural Audio Codeccited by 756
  7. 2019Linear Mode Connectivity and the Lottery Ticket Hypothesiscited by 627
  8. 2016Ups and Downs: Modeling the Visual Evolution of Fashion Trends with One-Class Collaborative Filteringcited by 517
  9. 2021Sentence-T5: Scalable Sentence Encoders from Pre-trained Text-to-Text Modelscited by 462
  10. 2024Model Merging in LLMs, MLLMs, and Beyond: Methods, Theories, Applications and Opportunitiescited by 247
  11. 2022Transformer Memory as a Differentiable Search Indexcited by 242
  12. 2023Recommender Systems with Generative Retrievalcited by 191
12 more
  1. 2024MagMax: Leveraging Model Merging for Seamless Continual Learningcited by 67
  2. 2023Learning to Tokenize for Generative Retrievalcited by 66
  3. 2023Knowledge is a Region in Weight Space for Fine-tuned Language Modelscited by 58
  4. 2023An Empirical Study of Multimodal Model Mergingcited by 52
  5. 2023Weighted Ensemble Models Are Strong Continual Learnerscited by 41
  6. 2024LiNeS: Post-training Layer Scaling Prevents Forgetting and Enhances Model Mergingcited by 39
  7. 2023Early Weight Averaging meets High Learning Rates for LLM Pre-trainingcited by 36
  8. 2024Mitigating Catastrophic Forgetting in Language Transfer via Model Mergingcited by 36
  9. 2024How to Merge Your Multimodal Models Over Time?cited by 23
  10. 2024DAM: Dynamic Adapter Merging for Continual Video QA Learningcited by 18
  11. 2025Soup to go: mitigating forgetting during continual learning with model averagingcited by 11
  12. 2025Continual Learning in Vision-Language Models via Aligned Model Mergingcited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026LLMs Need Encoders for Semantic IDs Toocited by 2

Similar, not cited

Close in wording, no citation either way

  1. 2025Catastrophic Forgetting in LLMs: A Comparative Analysis Across Language Tasks
  2. 2025RECALL: REpresentation-aligned Catastrophic-forgetting ALLeviation via Hierarchical Model Merging
  3. 2024Learning From Failure: Integrating Negative Examples when Fine-tuning Large Language Models as Agents
  4. 2026Adaptive Retrieval helps Reasoning in LLMs -- but mostly if it's not used
  5. 2026Retrieval-Augmented LLM Agents: Learning to Learn from Experience
  6. 2024From Artificial Needles to Real Haystacks: Improving Retrieval Capabilities in LLMs by Finetuning on Synthetic Data
  7. 2024Balancing Speciality and Versatility: A Coarse to Fine Framework for Mitigating Catastrophic Forgetting in Large Language Models
  8. 2026Tool-Adaptive LLM Reranker