mlsanityData to 7 Oct 2026

Graded Entity-Familiarity Readouts in Language Models: Polish Adaptation, Cross-Language Robustness, and Refusal Steering

arXiv 2607.13568 ↗First posted 15 Jul 2026cs.CL cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Can a language model estimate its familiarity with an entity before generating an answer? We study activations at the final prompt token in twelve instruction-tuned models from the Bielik, PLLuM, Gemma-4, and Qwen3 families, using a new dataset of 1,440 Polish entities spanning four domains and ten Wikipedia-pageview deciles, plus fabricated controls. Familiarity-probe scores separate real from fabricated entities in every family; in the Polish-adapted Bielik and PLLuM families they additionally track entity popularity (model-mean Spearman ρ 0.28-0.57, versus at most 0.11 in Gemma-4 and Qwen3), a pattern more strongly associated with Polish adaptation than with parameter count in this model sample. In a paired experiment on two families, probes retain 96-101% of within-language AUROC when the Polish question stem is replaced with an English one around unchanged entity names, showing robustness to prompt language in this setting. In Gemma-4-12B, the only model that natively refuses, adding a one-dimensional familiarity direction at a single layer moves refusal rates monotonically in both directions (0.24 to 1.00 on well-known entities; 0.73 to 0.00 on unknown ones). Finally, a calibrated familiarity probe is competitive among pre-generation abstention gates, although post-generation detectors better predict behavioral error on average. These results support a graded pre-generation entity-familiarity readout, and a separation between representational familiarity and the policy that converts it into abstention.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 38 of its 45 references that are in the map. First posted on arXiv 15 Jul 2026.

Builds on

38 of 45 references.

References · 38

  1. 2017On Calibration of Modern Neural Networkscited by 4,860
  2. 2023Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflectioncited by 1,800
  3. 2022Language Models (Mostly) Know What They Knowcited by 1,783
  4. 2023Inference-Time Intervention: Eliciting Truthful Answers from a Language Modelcited by 1,259
  5. 2022When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memoriescited by 1,103
  6. 2023Steering Llama 2 via Contrastive Activation Additioncited by 1,009
  7. 2024Refusal in Language Models Is Mediated by a Single Directioncited by 960
  8. 2023Steering Language Models With Activation Engineeringcited by 824
  9. 2017Selective Classification for Deep Neural Networkscited by 799
  10. 2023Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedbackcited by 788
  11. 2022Discovering Latent Knowledge in Language Models Without Supervisioncited by 733
  12. 2022Teaching Models to Express Their Uncertainty in Wordscited by 731
26 more
  1. 2023The Geometry of Truth: Emergent Linear Structure in Large Language Model Representations of True/False Datasetscited by 673
  2. 2023The Internal State of an LLM Knows When It's Lyingcited by 644
  3. 2022Large Language Models Struggle to Learn Long-Tail Knowledgecited by 452
  4. 2024INSIDE: LLMs' Internal States Retain the Power of Hallucination Detectioncited by 317
  5. 2024Do Llamas Work in English? On the Latent Language of Multilingual Transformerscited by 275
  6. 2024Semantic Entropy Probes: Robust and Cheap Hallucination Detection in LLMscited by 246
  7. 2024LLMs Know More Than They Show: On the Intrinsic Representation of LLM Hallucinationscited by 230
  8. 2024Programming Refusal with Conditional Activation Steeringcited by 191
  9. 2024Do I Know This Entity? Knowledge Awareness and Hallucinations in Language Modelscited by 114
  10. 2024HaloScope: Harnessing Unlabeled LLM Generations for Hallucination Detectioncited by 96
  11. 2023On Early Detection of Hallucinations in Factual Question Answeringcited by 54
  12. 2025Steer LLM Latents for Hallucination Detectioncited by 49
  13. 2025Hallucination Detection in LLMs Using Spectral Features of Attention Mapscited by 42
  14. 2024Estimating Knowledge in Large Language Models Without Generating a Single Tokencited by 39
  15. 2025No Answer Needed: Predicting LLM Answer Accuracy from Question-Only Linear Probescited by 35
  16. 2025Real-Time Detection of Hallucinated Entities in Long-Form Generationcited by 33
  17. 2025SemEval-2025 Task 3: Mu-SHROOM, the Multilingual Shared Task on Hallucinations and Related Observable Overgeneration Mistakescited by 27
  18. 2025On Linear Representations and Pretraining Data Frequency in Language Modelscited by 23
  19. 2025EigenTrack: Spectral Activation Feature Tracking for Hallucination and Out-of-Distribution Detection in LLMs and VLMscited by 10
  20. 2025PLLuM: A Family of Polish Large Language Modelscited by 7
  21. 2025Hallucination reduction with CASAL: Contrastive Activation Steering For Amortized Learningcited by 6
  22. 2025D2HScore: Reasoning-Aware Hallucination Detection via Semantic Breadth and Depth Analysis in LLMscited by 5
  23. 2026MultiHaluDet: Multilingual Hallucination Detection via LLM Hidden State Probingcited by 5
  24. 2026Bielik-Minitron-7B: Compressing Large Language Models via Structured Pruning and Knowledge Distillation for the Polish Languagecited by 4
  25. 2026Pretraining Exposure Explains Popularity Judgments in Large Language Modelscited by 1
  26. 2026Does Bielik Know What It Doesn't Know? Activation Dispersion Separates Entity Familiarity from Factual Reliability Across Model Scalecited by 1

Built on by

Not cited yet.