mlsanityData to 14 Oct 2026

Conflict-Free Replicated Data Types for Neural Network Model Merging: A Two-Layer Architecture Enabling CRDT-Compliant Model Merging Across 26 Strategies

arXiv 2605.19373 ↗First posted 16 May 2026cs.DC cs.AI cs.LGcited by 2

Not on the map. Why?

Abstract (arXiv)

All 26 neural network merge strategies we tested including weight averaging, SLERP, TIES, DARE, Fisher merging, and evolutionary approaches -- fail the algebraic properties (commutativity, associativity, idempotency) required for conflict-free distributed operation. We prove that this failure is structural: normalisation-based merges cannot simultaneously satisfy all three properties. To resolve this, we present a two-layer architecture -- CRDTMergeState -- that wraps any merge strategy in a CRDT-compliant (Conflict-Free Replicated Data Type) layer. Layer 1 manages contributions via OR-Set CRDT semantics, where the merge operation is set union -- trivially commutative, associative, and idempotent. Layer 2 applies merge strategies as deterministic pure functions over a canonically-ordered contribution set, with randomness seeded from the Merkle root. We prove that this separation guarantees Strong Eventual Consistency: all replicas receiving the same contributions compute identical merged models, regardless of message ordering. Empirical validation spans three tiers: controlled 4x4 tensors (104/104 tests pass), production-scale models up to 7.24B parameters (208 strategy-level tests, 43,368 layer-level property checks at capped tensor resolution), and multi-node convergence under gossip and partition healing (100 nodes, 20 orderings), with CRDT overhead below 0.5 ms. Because the wrapper is transparent, downstream performance is identical by construction, confirmed via byte-identical output verification. The reference implementation is available as crdt-merge v0.9.4.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 23 of its 41 references that are in the map. First posted on arXiv 16 May 2026.

Builds on

23 of 41 references.

References · 23

  1. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,669
  2. 2018Federated Optimization in Heterogeneous Networkscited by 2,987
  3. 2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,396
  4. 2022Editing Models with Task Arithmeticcited by 1,120
  5. 2023TIES-Merging: Resolving Interference When Merging Modelscited by 818
  6. 2023Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunchcited by 694
  7. 2021Merging Models with Fisher-Weighted Averagingcited by 621
  8. 2017Can Decentralized Algorithms Outperform Centralized Algorithms? A Case Study for Decentralized Parallel Stochastic Gradient Descentcited by 502
  9. 2022Dataless Knowledge Fusion by Merging Weights of Language Modelscited by 387
  10. 2024Model Merging in LLMs, MLLMs, and Beyond: Methods, Theories, Applications and Opportunitiescited by 254
  11. 2023AdaMerging: Adaptive Model Merging for Multi-Task Learningcited by 252
  12. 2019Decentralized Stochastic Optimization and Gossip Algorithms with Compressed Communicationcited by 221
11 more
  1. 2024Evolutionary Optimization of Model Merging Recipescited by 208
  2. 2024Arcee's MergeKit: A Toolkit for Merging Large Language Modelscited by 194
  3. 2023Model Breadcrumbs: Scaling Multi-Task Model Merging with Sparse Maskscited by 133
  4. 2024EMR-Merging: Tuning-Free High-Performance Model Mergingcited by 110
  5. 2024Representation Surgery for Multi-Task Model Mergingcited by 96
  6. 2024DELLA-Merging: Reducing Interference in Model Merging through Magnitude-Based Samplingcited by 81
  7. 2023Git-Theta: A Git Extension for Collaborative Development of Machine Learning Modelscited by 15
  8. 2016A Conflict-Free Replicated JSON Datatypecited by 6
  9. 2016Delta State Replicated Data Typescited by 4
  10. 2020Merkle-CRDTs: Merkle-DAGs meet CRDTscited by 2
  11. 2018Conflict-free Replicated Data Types (CRDTs)cited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.