mlsanityData to 14 Oct 2026

Omega-S: A Functional Resilience Index for LLM Fine-Tuning

arXiv 2608.03887 ↗First posted 4 Aug 2026cs.LG cs.NE q-bio.MNcited by 0

Not on the map. Why?

Abstract (arXiv)

Fine-tuning a large language model on new data degrades what it previously learned. We present Omega-S, a drop-in penalty computed from the weight matrix alone: it needs no previous-task data, no Fisher matrix and no stored copy of the old weights. It is three lines in an existing training loop and adds under 4% to the cost of a step. Retention. On Llama-3-8B with LoRA, fine-tuned from code to prose and measured by HumanEval over ten seeds, Omega-S retains more of the original capability than no regularisation on 9 of 10 seeds (0.173 -> 0.238 absolute pass@1; sign test one-sided p=0.011, Wilcoxon p=0.006), as a retention ratio, 62.9% -> 84.1%. It also beats tuned weight decay on 10 of 10 seeds (p=0.002) and tuned EWC on 8 of 10 (p=0.014), every arm re-measured in the same session. Mechanism, measured rather than asserted. Omega-S is topological by construction, its objective built from Tr(A^3), but we measured which of its four factors actually moves and three do not: their elasticity with respect to the weights is at or below 1e-4, against 9e-3 for the degree-variance term. As implemented, the composite reduces to a penalty on the variance of node degrees, which means row magnitude in square modules and directional alignment in non-square ones. We report this because a method whose name promises one thing and whose gradient does another should say so. We also enumerate the open design choices, including a contrast-preserving construction that does what it was designed to do and makes retention worse on all ten seeds. Repeating an identical configuration, same seed and same hardware, gives a standard deviation of 0.104 in retention ratio. We have not found this quantified for low-rank fine-tuning of language models, and it bounds every seed-paired comparison in this literature, ours included. Code, per-seed results and the full record of negative results are available.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 31 of its 41 references that are in the map. First posted on arXiv 4 Aug 2026.

Builds on

31 of 41 references.

References · 31

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  3. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  4. 2016Overcoming catastrophic forgetting in neural networkscited by 4,787
  5. 2016Pointer Sentinel Mixture Modelscited by 3,096
  6. 2016Learning without Forgettingcited by 2,496
  7. 2020Sharpness-Aware Minimization for Efficiently Improving Generalizationcited by 1,417
  8. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,386
  9. 2023SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shotcited by 1,121
  10. 2022LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scalecited by 906
  11. 2023PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallelcited by 660
  12. 2019Orthogonal Gradient Descent for Continual Learningcited by 369
19 more
  1. 2023Orthogonal Subspace Learning for Language Model Continual Learningcited by 312
  2. 2023Outlier Weighed Layerwise Sparsity (OWL): A Missing Secret Sauce for Pruning LLMs to High Sparsitycited by 165
  3. 2023TRACE: A Comprehensive Benchmark for Continual Learning in Large Language Modelscited by 66
  4. 2020Hutch++: Optimal Stochastic Trace Estimationcited by 63
  5. 2024AlphaPruning: Using Heavy-Tailed Self Regularization Theory for Improved Layer-wise Pruning of Large Language Modelscited by 36
  6. 2016A topological insight into restricted Boltzmann machinescited by 32
  7. 2021On the Reproducibility of Neural Network Predictionscited by 29
  8. 2018Deterministic Implementations for Reproducibility in Deep Reinforcement Learningcited by 19
  9. 2025DarwinLM: Evolutionary Structured Pruning of Large Language Modelscited by 18
  10. 2025Sculpting Subspaces: Constrained Full Fine-Tuning in LLMs for Continual Learningcited by 15
  11. 2026Mechanistic Analysis of Catastrophic Forgetting in Large Language Models During Continual Fine-tuningcited by 14
  12. 2020Non-Determinism in TensorFlow ResNetscited by 11
  13. 2020Fiedler Regularization: Learning Neural Networks with Graph Sparsitycited by 8
  14. 2025Efficient Shapley Value-based Non-Uniform Pruning of Large Language Modelscited by 8
  15. 2026Muon-OGD: Muon-based Spectral Orthogonal Gradient Projection for LLM Continual Learningcited by 5
  16. 2026Spectral Manifold Regularization for Stable and Modular Routing in Deep MoE Architecturescited by 4
  17. 2026Subspace Geometry Governs Catastrophic Forgetting in Low-Rank Adaptationcited by 4
  18. 2026Catastrophic Forgetting is Low-Rank: A Function-Space Theory for Continual Adaptationcited by 2
  19. 2026From Weights to Features: SAE-Guided Activation Regularization for LLM Continual Learningcited by 2

Built on by

Not cited yet.