# Weight Oracles: Reading Neural Network Weights with Language Models

arXiv: https://arxiv.org/abs/2610.07334 · HTML: https://mlsanity.com/paper/2610.07334/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Backdoor and data poisoning attacks, one of 12 directions in LLM safety and security.
- Builds on 13 of its 24 references that are in the map.
- First posted on arXiv 5 Oct 2026.

## Placed in

- LLM safety and security / [Backdoor and data poisoning attacks](https://mlsanity.com/safety/backdoor-and-data-poisoning-attacks/)

## Builds on (most cited first, 13 of 13)

- [BadNets: Identifying Vulnerabilities in the Machine Learning Model Supply Chain](https://mlsanity.com/paper/1708.06733/) (2017)
- [Progress measures for grokking via mechanistic interpretability](https://mlsanity.com/paper/2301.05217/) (2023)
- [Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasets](https://mlsanity.com/paper/2201.02177/) (2022)
- [Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Training](https://mlsanity.com/paper/2401.05566/) (2024)
- [Activation Oracles: Training and Evaluating LLMs as General-Purpose Activation Explainers](https://mlsanity.com/paper/2512.15674/) (2025)
- [LatentQA: Teaching LLMs to Decode Activations Into Natural Language](https://mlsanity.com/paper/2412.08686/) (2024)
- [Learning to Interpret Weight Differences in Language Models](https://mlsanity.com/paper/2510.05092/) (2025)
- [Position: Weight Space Should Be a First-Class Generative AI Modality](https://mlsanity.com/paper/2605.18632/) (2026)
- [Unelicitable Backdoors in Language Models via Cryptographic Transformer Circuits](https://mlsanity.com/paper/2406.02619/) (2024)
- [LoRA as Oracle](https://mlsanity.com/paper/2601.11207/) (2026)
- [Solving Trojan Detection Competitions with Linear Weight Classification](https://mlsanity.com/paper/2411.03445/) (2024)
- [Weight space Detection of Backdoors in LoRA Adapters](https://mlsanity.com/paper/2602.15195/) (2026)
- [Z-PEFT: Zero-shot Backdoor Detection in Parameter-Efficient Fine-Tuning via Canonical Spectral Signatures](https://mlsanity.com/paper/2608.02271/) (2026)

## Abstract (arXiv)

Interpretability methods for neural networks are predominantly reactive: they analyse activations produced during specific forward passes, requiring known inputs to find hidden capabilities such as backdoors. We propose Weight Oracles, fine-tuned language models that diagnose properties of a target network by reading its raw weights directly, without behavioural testing. We investigate this paradigm in two phases. Phase I establishes feasibility: through a staged curriculum and an external chain-of-computation that delegates parameter-free operations to deterministic code, an explainer LLM learns to simulate the forward pass of small transformers from their weights, achieving 99% holdout accuracy on unseen targets. Phase II repurposes this infrastructure for safety auditing. We train an oracle on natural language diagnostic questions about weight anomalies using only benign pathologies as training signal, and evaluate it zero-shot on backdoors absent from training. The oracle achieves AUROC 0.93 on attention-routed backdoors and 0.81 across a diversified threat distribution including stealth and adversarially regularized variants. Hand-crafted statistical detectors are sharp on the threat models they implicitly target but collapse on threat-model shift, while the oracle remains uniformly competent across attack types. Scaling to realistic model sizes remains the principal open challenge.