mlsanityData to 7 Oct 2026

Challenges with unsupervised LLM knowledge discovery

arXiv 2312.10029 ↗First posted 15 Dec 2023cs.LG cs.AIcited by 30

Not on the map. Why?

Abstract (arXiv)

We show that existing unsupervised methods on large language model (LLM) activations do not discover knowledge -- instead they seem to discover whatever feature of the activations is most prominent. The idea behind unsupervised knowledge elicitation is that knowledge satisfies a consistency structure, which can be used to discover knowledge. We first prove theoretically that arbitrary features (not just knowledge) satisfy the consistency structure of a particular leading unsupervised knowledge-elicitation method, contrast-consistent search (Burns et al. - arXiv:2212.03827). We then present a series of experiments showing settings in which unsupervised methods result in classifiers that do not predict knowledge, but instead predict a different prominent feature. We conclude that existing unsupervised methods for discovering latent knowledge are insufficient, and we contribute sanity checks to apply to evaluating future knowledge elicitation methods. Conceptually, we hypothesise that the identification issues explored here, e.g. distinguishing a model's knowledge from that of a simulated character's, will persist for future unsupervised methods.

Key facts. Cited by 30 papers in the map (data through 7 Oct 2026). Builds on 31 of its 41 references that are in the map. First posted on arXiv 15 Dec 2023.

Builds on

31 of 41 references.

References · 31

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  5. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  6. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  7. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
  8. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,579
  9. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,140
  10. 2016Understanding intermediate layers using linear classifier probescited by 1,279
  11. 2023Inference-Time Intervention: Eliciting Truthful Answers from a Language Modelcited by 1,259
  12. 2023Representation Engineering: A Top-Down Approach to AI Transparencycited by 1,239
19 more
  1. 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,023
  2. 2018Challenging Common Assumptions in the Unsupervised Learning of Disentangled Representationscited by 929
  3. 2022Toy Models of Superpositioncited by 928
  4. 2022Discovering Latent Knowledge in Language Models Without Supervisioncited by 733
  5. 2023The Geometry of Truth: Emergent Linear Structure in Large Language Model Representations of True/False Datasetscited by 673
  6. 2023The Internal State of an LLM Knows When It's Lyingcited by 644
  7. 2023Eliciting Latent Predictions from Transformers with the Tuned Lenscited by 579
  8. 2023Role-Play with Large Language Modelscited by 290
  9. 2023AI Deception: A Survey of Examples, Risks, and Potential Solutionscited by 185
  10. 2022Talking About Large Language Modelscited by 174
  11. 2021Alignment of Language Agentscited by 167
  12. 2023Large Language Models can Strategically Deceive their Users when Put Under Pressurecited by 132
  13. 2021Truthful AI: Developing and governing AI that does not liecited by 109
  14. 2023How to Catch an AI Liar: Lie Detection in Black-Box LLMs by Asking Unrelated Questionscited by 82
  15. 2023Still No Lie Detector for Language Models: Probing Empirical and Conceptual Roadblockscited by 77
  16. 2022Discovering Latent Concepts Learned in BERTcited by 56
  17. 2023Gaussian Process Probes (GPP) for Uncertainty-Aware Probingcited by 11
  18. 2023Generalization Analogies: A Testbed for Generalizing AI Oversight to Hard-To-Measure Domainscited by 8
  19. 2023Comparing Optimization Targets for Contrast-Consistent Searchcited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 16; 2025: 8; 2026: 6.

30 papers.

Sort:

By area:

  1. 2024Mechanistic Interpretability for AI Safety -- A Reviewcited by 415
  2. 2024Semantic Entropy Probes: Robust and Cheap Hallucination Detection in LLMscited by 246
  3. 2024Towards Principled Evaluations of Sparse Autoencoders for Interpretability and Controlcited by 83
  4. 2024On the Relationship between Truth and Political Bias in Language Modelscited by 31
  5. 2024Localizing Paragraph Memorization in Language Modelscited by 30
  6. 2024Standards for Belief Representations in LLMscited by 30
  7. 2024The Probabilities Also Matter: A More Faithful Metric for Faithfulness of Free-Text Explanations in Large Language Modelscited by 29
  8. 2025To Steer or Not to Steer? Mechanistic Error Reduction with Abstention for Language Modelscited by 26
  9. 2024Fundamental Problems With Model Editing: How Should Rational Belief Revision Work in LLMs?cited by 24
  10. 2024A Survey on the Honesty of Large Language Modelscited by 21
  11. 2024Holistic Safety and Responsibility Evaluations of Advanced AI Modelscited by 18
  12. 2025Unsupervised Elicitation of Language Modelscited by 11
18 more
  1. 2024Prospects for inconsistency detection using large language models and sheavescited by 10
  2. 2025Improving Preference Extraction In LLMs By Identifying Latent Knowledge Through Classifying Probescited by 9
  3. 2024Truth-value judgment in language models: 'truth directions' are context sensitivecited by 7
  4. 2024Defining Knowledge: Bridging Epistemology and Large Language Modelscited by 7
  5. 2025Watch the Weights: Unsupervised monitoring and control of fine-tuned LLMscited by 7
  6. 2024Enhanced Language Model Truthfulness with Learnable Intervention and Uncertainty Expressioncited by 6
  7. 2024Chinese SafetyQA: A Safety Short-form Factuality Benchmark for Large Language Modelscited by 6
  8. 2025Unboxing the Black Box: Mechanistic Interpretability for Algorithmic Understanding of Neural Networkscited by 6
  9. 2024Cluster-norm for Unsupervised Probing of Knowledgecited by 4
  10. 2025Representation-based Broad Hallucination Detectors Fail to Generalize Out of Distributioncited by 4
  11. 2026ContextRL: Enhancing MLLM's Knowledge Discovery Efficiency with Context-Augmented RLcited by 4
  12. 2025Efficient Knowledge Probing of Large Language Models by Adapting Pre-trained Embeddingscited by 3
  13. 2026Rift: A Conflict Signature for Deception in Language Modelscited by 1
  14. 2025LLM Probing with Contrastive Eigenproblems: Improving Understanding and Applicability of CCScited by 0
  15. 2026Unsupervised Elicitation of Moral Values from Language Modelscited by 0
  16. 2026The Truth Was Never Gone: Perfect Aliasing in Compliant-Context Truth Probescited by 0
  17. 2026A Lie Detector Test for Language Models: Reading Knowledge a Model Won't Revealcited by 0
  18. 2026Decodable In-Context State and Model Output Across Trainingcited by 0