mlsanityData to 7 Oct 2026

Polysemanticity or Polysemy? Lexical Identity Confounds Superposition Metrics

arXiv 2604.00443 ↗First posted 1 Apr 2026cs.CL cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

If the same neuron activates for both "lender" and "riverside," standard metrics attribute the overlap to superposition--the neuron must be compressing two unrelated concepts. This work explores how much of the overlap is due a lexical confound: neurons fire for a shared word form (such as "bank") rather than for two compressed concepts. A 2x2 factorial decomposition reveals that the lexical-only condition (same word, different meaning) consistently exceeds the semantic-only condition (different word, same meaning) across models spanning 110M-70B parameters. The confound carries into sparse autoencoders (18-36% of features blend senses), sits in <=1% of activation dimensions, and hurts downstream tasks: filtering it out improves word sense disambiguation and makes knowledge edits more selective (p = 0.002).

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 12 of its 29 references that are in the map. First posted on arXiv 1 Apr 2026.

Builds on

12 of 29 references.

References · 12

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  3. 2022Locating and Editing Factual Associations in GPTcited by 2,609
  4. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,734
  5. 2024Refusal in Language Models Is Mediated by a Single Directioncited by 960
  6. 2022Toy Models of Superpositioncited by 928
  7. 2022Transformer Feed-Forward Layers Build Predictions by Promoting Concepts in the Vocabulary Spacecited by 522
  8. 2018What you can cram into a single vector: Probing sentence embeddings for linguistic propertiescited by 452
  9. 2019Does BERT Make Any Sense? Interpretable Word Sense Disambiguation with Contextualized Embeddingscited by 55
  10. 2025Rethinking Evaluation of Sparse Autoencoders through the Representation of Polysemous Wordscited by 17
  11. 2020Analysis and Evaluation of Language Models for Word Sense Disambiguationcited by 6
  12. 2026Parameter-Efficient Token Embedding Editing for Clinical Class-Level Unlearningcited by 3

Built on by

Not cited yet.