Decomposing Evolutionary Mixture-of-LoRA Architectures: The Routing Lever, the Lifecycle Penalty, and a Substrate-Conditional Boundary
Not on the map. Why?
Abstract (arXiv)
We decompose an evolutionary mixture-of-LoRA system on a from-scratch ~150M-parameter widened-D substrate (D=1536, V=32000; D/V approx 0.048; the "widened-1536" substrate) into three factors -- a router rewrite (parallel sigmoid gate with learnable per-adapter floor and bounded temperature anneal, fed post-stack hidden states rather than token-embedding means), a per-domain leave-one-out evaluation scope, and a lifecycle of death plus alpha-blend inheritance plus SVD mutation plus slot reallocation -- and report a 5-of-8 partial 2^3 factorial run at n=3 seeds and 25000 adaptation steps per cell. The attribution chain is sharp on this substrate: the router rewrite carries the entire +0.0426 nat balanced log-PPL improvement (Delta = log PPL_ref - log PPL_test, positive = improvement; t=12.86, p=0.006) attributed to "the full evolutionary system vs the static B3 baseline"; the headline full-system-vs-B3 balanced contrast itself is +0.015 nats, t=1.94, p=0.19 at n=3 and does not clear alpha=0.05. The per-domain evaluation scope is null at seed-resolution, and the lifecycle is a net drag of approx -0.028 nats (t=-4.46,p=0.047 in the primary chain). An auxiliary alpha=0 inheritance counterfactual at n=3 seeds is sign-inconsistent at the headline metric and underpowered for either an equivalence or load-bearing conclusion (corrected from an earlier arithmetic-mean aggregator that erroneously cleared inheritance; see Appendix B.11). A base-perturbation probe directionally refutes a "genomic-context" reframe of the lifecycle role. A controllable synthetic sandbox locates a substrate-conditional regime boundary: evolutionary search on the routing channel is load-bearing only when adapters are pre-aligned to the task; in every other regime tested it underperforms, ties, or actively degrades the gradient solution.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 17 of its 26 references that are in the map. First posted on arXiv 11 May 2026.
Builds on
17 of 26 references.
References · 17
- 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
- 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
- 2023Large Language Models Cannot Self-Correct Reasoning Yetcited by 1,017
- 2022Mixture-of-Experts with Expert Choice Routingcited by 623
- 2017Evolution Strategies as a Scalable Alternative to Reinforcement Learningcited by 574
- 2017Population Based Training of Neural Networkscited by 367
- 2020Improving Reproducibility in Machine Learning Research (A Report from the NeurIPS 2019 Reproducibility Program)cited by 322
- 2024Evolutionary Optimization of Model Merging Recipescited by 208
- 2024MixLoRA: Enhancing Large Language Models Fine-Tuning with LoRA-based Mixture of Expertscited by 147
- 2025Nested Learning: The Illusion of Deep Learning Architecturescited by 72
- 2025Evolution Strategies at Scale: LLM Fine-Tuning Beyond Reinforcement Learningcited by 34
- 2017ES Is More Than Just a Traditional Finite-Difference Approximatorcited by 24
5 more
- 2025Heterogeneous Swarms: Jointly Optimizing Model Roles and Weights for Multi-LLM Systemscited by 23
- 2024Agent Skill Acquisition for Large Language Models via CycleQDcited by 12
- 2026The Blessing of Dimensionality in LLM Fine-tuning: A Variance-Curvature Perspectivecited by 10
- 2026Evolutionary Strategies lead to Catastrophic Forgetting in LLMscited by 7
- 2025Competition and Attraction Improve Model Fusioncited by 4
Built on by
Citing papers per year: 2026: 1.
1 paper.