mlsanityData to 7 Oct 2026

Does Bielik Know What It Doesn't Know? Activation Dispersion Separates Entity Familiarity from Factual Reliability Across Model Scale

arXiv 2607.07670 ↗First posted 8 Jul 2026cs.CL cs.LGcited by 1

Not on the map. Why?

Abstract (arXiv)

Large language models hallucinate most about entities they have never seen. We ask whether a model's activations betray entity familiarity before a single answer token is generated, and whether that signal predicts the factual reliability of the answers. On four Polish Bielik models (1.5B-11B parameters), we probe four entity domains (athletes, cities, writers, musicians), each with 42 well-known, 42 obscure-but-real, and 42 fabricated entities addressed by a one-sentence question (504 prompts per model). Two unsupervised, single-forward-pass dispersion measures over post-SwiGLU MLP activations, inverse participation ratio and spectral entropy, separate known from fabricated entities at AUROC 0.95-1.00 across all domains and scales; a supervised linear probe reaches 0.99-1.00. Both clear selection-aware permutation floors of about 0.70-0.74 (empirical p<=1e-3), survive held-out layer selection (0.93-0.99), and persist on real names (known vs. obscure-but-real: 0.96-1.00). The signal transfers across entity types (mean off-diagonal AUROC 0.92-0.99); a matched-template counterfactual shows the only large drops are template-caused, not entity-type effects, and the signal is diffuse across heads. This representational signal is already at ceiling at 1.5B, whereas behavioral factual reliability scales sharply: 0, 2, 10, and 19 of 42 known athletes are answered fully correctly by the 1.5B, 4.5B, 7B, and 11B models under a strict judge. Within known entities, separating correct from hallucinated answers is much harder (probe 0.93; dispersion no better than a first-token-entropy baseline). A five-sample semantic-entropy baseline reaches only 0.71-0.83 at 5x the inference cost. Despite this internal awareness, the models almost never abstain: an audit of 2,520 answers finds 2 refusals and 1 hedge. Entity familiarity and factual reliability are distinct phenomena on different scaling curves.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 25 of its 32 references that are in the map. First posted on arXiv 8 Jul 2026.

Builds on

25 of 32 references.

References · 25

  1. 2022When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memoriescited by 1,103
  2. 2023Semantic Uncertainty: Linguistic Invariances for Uncertainty Estimation in Natural Language Generationcited by 795
  3. 2023SelfCheckGPT: Zero-Resource Black-Box Hallucination Detection for Generative Large Language Modelscited by 787
  4. 2023The Internal State of an LLM Knows When It's Lyingcited by 644
  5. 2022Large Language Models Struggle to Learn Long-Tail Knowledgecited by 452
  6. 2024INSIDE: LLMs' Internal States Retain the Power of Hallucination Detectioncited by 317
  7. 2024Massive Activations in Large Language Modelscited by 277
  8. 2024Semantic Entropy Probes: Robust and Cheap Hallucination Detection in LLMscited by 246
  9. 2024LLMs Know More Than They Show: On the Intrinsic Representation of LLM Hallucinationscited by 230
  10. 2024Retrieval Head Mechanistically Explains Long-Context Factualitycited by 219
  11. 2024Compact Language Models via Pruning and Knowledge Distillationcited by 142
  12. 2024Lookback Lens: Detecting and Mitigating Contextual Hallucinations in Large Language Models Using Only Attention Mapscited by 140
13 more
  1. 2022The Lazy Neuron Phenomenon: On Emergence of Activation Sparsity in Transformerscited by 121
  2. 2024Do I Know This Entity? Knowledge Awareness and Hallucinations in Language Modelscited by 114
  3. 2024HaloScope: Harnessing Unlabeled LLM Generations for Hallucination Detectioncited by 96
  4. 2025Steer LLM Latents for Hallucination Detectioncited by 49
  5. 2025Hallucination Detection in LLMs Using Spectral Features of Attention Mapscited by 42
  6. 2025No Answer Needed: Predicting LLM Answer Accuracy from Question-Only Linear Probescited by 35
  7. 2025Real-Time Detection of Hallucinated Entities in Long-Form Generationcited by 33
  8. 2025SemEval-2025 Task 3: Mu-SHROOM, the Multilingual Shared Task on Hallucinations and Related Observable Overgeneration Mistakescited by 27
  9. 2025HalluVerse25: Fine-grained Multilingual Benchmark Dataset for LLM Hallucinationscited by 13
  10. 2025EigenTrack: Spectral Activation Feature Tracking for Hallucination and Out-of-Distribution Detection in LLMs and VLMscited by 10
  11. 2025D2HScore: Reasoning-Aware Hallucination Detection via Semantic Breadth and Depth Analysis in LLMscited by 5
  12. 2026MultiHaluDet: Multilingual Hallucination Detection via LLM Hidden State Probingcited by 5
  13. 2026Bielik-Minitron-7B: Compressing Large Language Models via Structured Pruning and Knowledge Distillation for the Polish Languagecited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.