Non-Interfering Weight Fields: Treating Model Parameters as a Continuously Extensible Function
Not on the map. Why?
Abstract (arXiv)
Large language models store all learned knowledge in a single, fixed weight vector. Teaching a model new capabilities requires modifying those same weights, inevitably degrading previously acquired knowledge. This fundamental limitation, known as catastrophic forgetting, has resisted principled solutions for decades. Existing approaches treat weights as immutable artifacts that must be protected through techniques like regularization heuristics, replay buffers, or isolated adapter modules. The problem is none of these provide a structural guarantee against forgetting. In this work, we propose Non-Interfering Weight Fields (NIWF), a framework that replaces the fixed weight paradigm with a learned function that generates weight configurations on demand from a continuous capability coordinate space. After training on a task, we commit the occupied coordinate region by snapshotting the fields outputs on anchor points to enforce a functional lock during all future training. We validate NIWF on sequential instructionfollowing and code generation tasks using Mistral-7B, demonstrating zero forgetting on committed tasks with competitive perplexity on new tasks. The framework introduces the notion of software-like versioning for neural network intelligence, where capabilities can be committed, extended, composed, and rolled back without retraining.
Key facts. No citing papers in the map yet (data through 13 Oct 2026). Builds on 21 of its 27 references that are in the map. First posted on arXiv 20 Feb 2026.
Builds on
21 of 27 references.
References · 21
- 2016Overcoming catastrophic forgetting in neural networkscited by 4,753
- 2014Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translationcited by 4,024
- 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,463
- 2023Mistral 7Bcited by 3,123
- 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,044
- 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,618
- 2016Progressive Neural Networkscited by 1,347
- 2017Memory Aware Synapses: Learning what (not) to forgetcited by 1,099
- 2018Experience Replay for Continual Learningcited by 1,082
- 2018Efficient Lifelong Learning with A-GEMcited by 1,053
- 2016HyperNetworkscited by 1,010
- 2020Dark Experience for General Continual Learning: a Strong, Simple Baselinecited by 901
9 more
- 2017PackNet: Adding Multiple Tasks to a Single Network by Iterative Pruningcited by 891
- 2024DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Modelscited by 853
- 2025Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attentioncited by 398
- 2023LoraHub: Efficient Cross-Task Generalization via Dynamic LoRA Compositioncited by 362
- 2023AdaLoRA: Adaptive Budget Allocation for Parameter-Efficient Fine-Tuningcited by 347
- 2025Gated Attention for Large Language Models: Non-linearity, Sparsity, and Attention-Sink-Freecited by 270
- 2019Continual learning with hypernetworkscited by 268
- 2023SwitchHead: Accelerating Transformers with Mixture-of-Experts Attentioncited by 34
- 2022Hyperdecoders: Instance-specific decoders for multi-task NLPcited by 21
Built on by
Not cited yet.