# Constructing Efficient Fact-Storing MLPs for Transformers

arXiv: https://arxiv.org/abs/2512.00207 · HTML: https://mlsanity.com/paper/2512.00207/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 28 Nov 2025.

## Built on by (most cited first, 3 of 3)

- [Universal Redundancies in Time Series Foundation Models](https://mlsanity.com/paper/2602.01605/) (2026)
- [Fine-Tuning and Evaluating Conversational AI for Agricultural Advisory](https://mlsanity.com/paper/2603.03294/) (2026)
- [Geometric Factual Recall in Transformers](https://mlsanity.com/paper/2605.12426/) (2026)

## Abstract (arXiv)

The success of large language models (LLMs) can be attributed in part to their ability to efficiently store factual knowledge as key-value mappings within their MLP parameters. Recent work has proposed explicit weight constructions to build such fact-storing MLPs, providing an improved understanding of LLM fact storage mechanisms. In this paper, we introduce an MLP construction framework that improves over previous constructions in three areas: it 1) works for all but a measure-zero set of feasible input-output pairs, 2) achieves asymptotically optimal parameter efficiency matching information-theoretic bounds for some embeddings, and 3) maintains usability within Transformers for factual recall. Through our improvements, we 1) discover a metric on value embeddings that characterizes facts-per-parameter scaling for both constructed and gradient-descent-trained MLPs, 2) identify a simple encoder-decoder mechanism that empirically matches gradient-descent MLP facts-per-parameter asymptotics across all the inputs and outputs we test, and 3) uncover a fundamental tradeoff between an MLP's fact-storage capacity and its usability within Transformers. Finally, we demonstrate a proof-of-concept application of fact-storing MLPs: modular fact editing on one-layer Transformers by textitreplacing entire MLPs at once.