mlsanityData to 7 Oct 2026

Cluster-norm for Unsupervised Probing of Knowledge

arXiv 2407.18712 ↗First posted 26 Jul 2024cs.AI cs.CL cs.LGcited by 4

Not on the map. Why?

Abstract (arXiv)

The deployment of language models brings challenges in generating reliable information, especially when these models are fine-tuned using human preferences. To extract encoded knowledge without (potentially) biased human labels, unsupervised probing techniques like Contrast-Consistent Search (CCS) have been developed (Burns et al., 2022). However, salient but unrelated features in a given dataset can mislead these probes (Farquhar et al., 2023). Addressing this, we propose a cluster normalization method to minimize the impact of such features by clustering and normalizing activations of contrast pairs before applying unsupervised probing techniques. While this approach does not address the issue of differentiating between knowledge in general and simulated knowledge - a major issue in the literature of latent knowledge elicitation (Christiano et al., 2021) - it significantly improves the ability of unsupervised probes to identify the intended knowledge amidst distractions.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Builds on 26 of its 38 references that are in the map. First posted on arXiv 26 Jul 2024.

Builds on

26 of 38 references.

References · 26

  1. 2023Mistral 7Bcited by 3,065
  2. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,579
  3. 2023Sparse Autoencoders Find Highly Interpretable Features in Language Modelscited by 1,359
  4. 2016Understanding intermediate layers using linear classifier probescited by 1,279
  5. 2023Inference-Time Intervention: Eliciting Truthful Answers from a Language Modelcited by 1,259
  6. 2023Representation Engineering: A Top-Down Approach to AI Transparencycited by 1,239
  7. 2023Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Promptingcited by 1,081
  8. 2021Ethical and social risks of harm from Language Modelscited by 957
  9. 2022Discovering Language Model Behaviors with Model-Written Evaluationscited by 805
  10. 2022Discovering Latent Knowledge in Language Models Without Supervisioncited by 733
  11. 2023The Geometry of Truth: Emergent Linear Structure in Large Language Model Representations of True/False Datasetscited by 673
  12. 2023The Internal State of an LLM Knows When It's Lyingcited by 644
14 more
  1. 2023Eliciting Latent Predictions from Transformers with the Tuned Lenscited by 579
  2. 2022Emergent World Representations: Exploring a Sequence Model Trained on a Synthetic Taskcited by 417
  3. 2023Emergent Linear Representations in World Models of Self-Supervised Sequence Modelscited by 313
  4. 2023Language Models Represent Space and Timecited by 306
  5. 2021Unsolved Problems in ML Safetycited by 291
  6. 2023AI Deception: A Survey of Examples, Risks, and Potential Solutionscited by 185
  7. 2023Explore, Establish, Exploit: Red Teaming Language Models from Scratchcited by 111
  8. 2021Truthful AI: Developing and governing AI that does not liecited by 109
  9. 2023Question Decomposition Improves the Faithfulness of Model-Generated Reasoningcited by 107
  10. 2024Identifying Functionally Important Features with End-to-End Sparse Dictionary Learningcited by 64
  11. 2023Eliciting Latent Knowledge from Quirky Language Modelscited by 56
  12. 2023Challenges with unsupervised LLM knowledge discoverycited by 30
  13. 2023Localizing Lying in Llama: Understanding Instructed Dishonesty on True-False Questions Through Prompting, Probing, and Patchingcited by 24
  14. 2024Using Degeneracy in the Loss Landscape for Mechanistic Interpretabilitycited by 12

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 3; 2026: 1.

4 papers.

  1. 2025Improving Preference Extraction In LLMs By Identifying Latent Knowledge Through Classifying Probescited by 9
  2. 2025Risk Assessment Framework for Code LLMs via Leveraging Internal Statescited by 8
  3. 2025KScope: A Framework for Characterizing the Knowledge Status of Language Modelscited by 3
  4. 2026Exploring Approaches for Detecting Memorization of Recommender System Data in Large Language Modelscited by 0