Beyond the Parameter Monolith: Reconstructive Memories, Executable Skills, and Residual Assembly for Language Models
Not on the map. Why?
Abstract (arXiv)
Language-model systems can separate contextual computation, persistent storage, and exact execution instead of updating all capabilities through one shared parameter system. We investigate FEM-ASM, a finite-element-method-inspired organization in which independently constructed document states and deterministic executable skills contribute typed proposals to a shared language-model state. An explicit residual operator reconciles proposals attached to common interface nodes. We evaluate this organization through controlled experiments and negative results rather than claiming a physical finite-element formulation of language. An attention-free Multi-Mesh prototype learns causal language modeling but does not establish competitive general capability. A versioned store contains 52,809 reconstructive memory elements near a 1.7-billion-floating-value budget; reconstruction is incomplete, with approximately 75% token accuracy. Support-aware lexical indices make these elements addressable under provenance-controlled query construction. For executable arithmetic, positional result observations substantially improve neural rendering relative to a repeated global result vector, and output substitutions change the model's preferred answer. A bounded attachment demonstration further measures the effect of making selected evidence available, without establishing the utility of loading an entire multi-billion-value store. The results support a separation of storage, execution, and neural coordination, while identifying unresolved limitations in question-only retrieval, unrestricted answer generation, and end-to-end efficiency.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 21 of its 27 references that are in the map. First posted on arXiv 2 Oct 2026.
Builds on
21 of 27 references.
References · 21
- 2017Attention Is All You Needcited by 60,361
- 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
- 2020Measuring Massive Multitask Language Understandingcited by 7,963
- 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
- 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,718
- 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,182
- 2016Pointer Sentinel Mixture Modelscited by 3,096
- 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,883
- 2017Deep Setscited by 1,421
- 2021Improving language models by retrieving from trillions of tokenscited by 1,293
- 2024The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scalecited by 1,160
- 2016The LAMBADA dataset: Word prediction requiring a broad discourse contextcited by 992
9 more
- 2019Generalization through Memorization: Nearest Neighbor Language Modelscited by 800
- 2019Compressive Transformers for Long-Range Sequence Modellingcited by 726
- 2018Universal Transformerscited by 647
- 2019Deep Equilibrium Modelscited by 615
- 2023Hyena Hierarchy: Towards Larger Convolutional Language Modelscited by 393
- 2025SmolLM2: When Smol Goes Big -- Data-Centric Training of a Small Language Modelcited by 335
- 2022Memorizing Transformerscited by 226
- 2022Language Modelling with Pixelscited by 55
- 2025Emergent Semantics Beyond Token Embeddings: Transformer LMs with Frozen Visual Unicode Representationscited by 5
Built on by
Not cited yet.