# Signal in the Noise: Polysemantic Interference Transfers and Predicts Cross-Model Influence arXiv: https://arxiv.org/abs/2505.11611 ยท HTML: https://mlsanity.com/paper/2505.11611/ ## Key facts - Cited by 4 papers in the map (data through 14 Oct 2026). - Placed in SAE features and their applications, one of 13 directions in Mechanistic interpretability. - Builds on 30 of its 43 references that are in the map. - First posted on arXiv 16 May 2025. ## Placed in - Mechanistic interpretability / [SAE features and their applications](https://mlsanity.com/mechinterp/sae-features-and-their-applications/) ## Builds on (most cited first, 30 of 30) - [Universal and Transferable Adversarial Attacks on Aligned Language Models](https://mlsanity.com/paper/2307.15043/) (2023) - [Enriching Word Vectors with Subword Information](https://mlsanity.com/paper/1607.04606/) (2016) - [Sparse Autoencoders Find Highly Interpretable Features in Language Models](https://mlsanity.com/paper/2309.08600/) (2023) - [Steering Llama 2 via Contrastive Activation Addition](https://mlsanity.com/paper/2312.06681/) (2023) - [Scaling and evaluating sparse autoencoders](https://mlsanity.com/paper/2406.04093/) (2024) - [Sparse Feature Circuits: Discovering and Editing Interpretable Causal Graphs in Language Models](https://mlsanity.com/paper/2403.19647/) (2024) - [Mechanistic Interpretability for AI Safety -- A Review](https://mlsanity.com/paper/2404.14082/) (2024) - [Jumping Ahead: Improving Reconstruction Fidelity with JumpReLU Sparse Autoencoders](https://mlsanity.com/paper/2407.14435/) (2024) - [Transcoders Find Interpretable LLM Feature Circuits](https://mlsanity.com/paper/2406.11944/) (2024) - [A is for Absorption: Studying Feature Splitting and Absorption in Sparse Autoencoders](https://mlsanity.com/paper/2409.14507/) (2024) - [Backdoor Defense via Decoupling the Training Process](https://mlsanity.com/paper/2202.03423/) (2022) - [Sparse Autoencoders Trained on the Same Data Learn Different Features](https://mlsanity.com/paper/2501.16615/) (2025) - [A Primer on the Inner Workings of Transformer-based Language Models](https://mlsanity.com/paper/2405.00208/) (2024) - [A Survey on Sparse Autoencoders: Interpreting the Internal Mechanisms of Large Language Models](https://mlsanity.com/paper/2503.05613/) (2025) - [Composite Backdoor Attacks Against Large Language Models](https://mlsanity.com/paper/2310.07676/) (2023) - [Applying sparse autoencoders to unlearn knowledge in language models](https://mlsanity.com/paper/2410.19278/) (2024) - [Shared Global and Local Geometry of Language Model Embeddings](https://mlsanity.com/paper/2503.21073/) (2025) - [Don't trust your eyes: on the (un)reliability of feature visualizations](https://mlsanity.com/paper/2306.04719/) (2023) - [PURE: Turning Polysemantic Neurons Into Pure Features by Identifying Relevant Circuits](https://mlsanity.com/paper/2404.06453/) (2024) - [SAEs Can Improve Unlearning: Dynamic Sparse Autoencoder Guardrails for Precision Unlearning in LLMs](https://mlsanity.com/paper/2504.08192/) (2025) - [Scaling Laws for Adversarial Attacks on Language Model Activations](https://mlsanity.com/paper/2312.02780/) (2023) - [What Causes Polysemanticity? An Alternative Origin Story of Mixed Selectivity from Incidental Causes](https://mlsanity.com/paper/2312.03096/) (2023) - [Linear Explanations for Individual Neurons](https://mlsanity.com/paper/2405.06855/) (2024) - [Understanding polysemanticity in neural networks through coding theory](https://mlsanity.com/paper/2401.17975/) (2024) - [Layers at Similar Depths Generate Similar Activations Across LLM Architectures](https://mlsanity.com/paper/2504.08775/) (2025) - [Don't Forget It! Conditional Sparse Autoencoder Clamping Works for Unlearning](https://mlsanity.com/paper/2503.11127/) (2025) - [Learning from Emergence: A Study on Proactively Inhibiting the Monosemantic Neurons of Artificial Neural Networks](https://mlsanity.com/paper/2312.11560/) (2023) - [Tackling Polysemanticity with Neuron Embeddings](https://mlsanity.com/paper/2411.08166/) (2024) - [SAFR: Neuron Redistribution for Interpretability](https://mlsanity.com/paper/2501.16374/) (2025) - [OASIS: Order-Augmented Strategy for Improved Code Search](https://mlsanity.com/paper/2503.08161/) (2025) ## Built on by (most cited first, 4 of 4) - [The Achilles' Heel of LLMs: How Altering a Handful of Neurons Can Cripple Language Abilities](https://mlsanity.com/paper/2510.10238/) (2025) - [Theory of Mind and Self-Attributions of Mentality are Dissociable in LLMs](https://mlsanity.com/paper/2603.28925/) (2026) - [From Concept-Aligned Tokens to Vulnerable Features: Mechanistic Localization of Jailbreaks](https://mlsanity.com/paper/2604.23130/) (2026) - [Interference Beyond Geometry in Concept Extraction](https://mlsanity.com/paper/2609.35351/) (2026) ## Abstract (arXiv) Polysemanticity is pervasive in language models and remains a major challenge for interpretation and model behavioral control. Leveraging sparse autoencoders (SAEs), we map the polysemantic topology of two small models (Pythia-70M and GPT-2-Small) to identify SAE feature pairs that are semantically unrelated yet exhibit interference within models. We intervene at four foci (prompt, token, feature, neuron) and measure induced shifts in the next-token prediction distribution, uncovering polysemantic structures that expose a systematic vulnerability in these models. Critically, interventions distilled from counterintuitive interference patterns shared by two small models transfer reliably to larger instruction-tuned models (Llama-3.1-8B/70B-Instruct and Gemma-2-9B-Instruct), yielding predictable behavioral shifts without access to model internals. These findings challenge the view that polysemanticity is purely stochastic, demonstrating instead that interference structures generalize across scale and family. Such generalization suggests a convergent, higher-order organization of internal representations, which is only weakly aligned with intuition and structured by latent regularities, offering new possibilities for both black-box control and theoretical insight into human and artificial cognition.