Does the Truthfulness Signal Survive Code-Mixing? Probing Hidden States for Hallucination Detection in Hinglish
Not on the map. Why?
Abstract (arXiv)
Hidden-state hallucination probing - training a linear classifier on an LLM's internal activations to detect whether a generated answer is faithful to the input - is an active area of 2026 research, with recent work reporting 0.90-1.00 AUROC across several benchmarks and languages. However, none of this work has tested probes on code-mixed input, despite the fact that a huge population of chatbot users write in Hindi-English code-mixed text ("Hinglish"). We address this gap directly: does a hallucination probe trained on clean-language hidden states transfer to Hinglish, or does the signal degrade under code-mixing? We construct a 5,674-item Hindi/English/Hinglish QA benchmark, generate and label 17,022 model responses across three open-weight 7-8B LLMs (Qwen2.5-7B, Mistral-7B, Llama-3.1-8B), extract per-layer hidden states at two token positions, and train linear and MLP probes for in-distribution detection and cross-lingual transfer. We find that the hallucination signal survives code-mixing well: transfer AUROC ranges from 0.88 to 0.99, with gaps of mostly under 0.05 AUROC relative to in-distribution performance, and that Hindi-trained probes transfer to Hinglish more reliably than English-trained probes. As an independent, practically motivated finding, all three models hallucinate substantially more on Hindi and Hinglish than on English for matched facts. We release our code and synthetic Hinglish QA dataset to support further work on code-mixed hallucination detection.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 11 of its 15 references that are in the map. First posted on arXiv 25 Aug 2026.
Builds on
11 of 15 references.
References · 11
- 2024The Llama 3 Herd of Modelscited by 14,715
- 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,148
- 2023The Internal State of an LLM Knows When It's Lyingcited by 659
- 2019On the Cross-lingual Transferability of Monolingual Representationscited by 578
- 2017Word Translation Without Parallel Datacited by 487
- 2024LLMs Know More Than They Show: On the Intrinsic Representation of LLM Hallucinationscited by 244
- 2020GLUECoS : An Evaluation Benchmark for Code-Switched NLPcited by 102
- 2022Towards Leaving No Indic Language Behind: Building Monolingual Corpora, Benchmark and Models for Indic Languagescited by 102
- 2020LinCE: A Centralized Benchmark for Linguistic Code-switching Evaluationcited by 93
- 2026MultiHaluDet: Multilingual Hallucination Detection via LLM Hidden State Probingcited by 5
- 2026Prompt Embedding Probes (PEP): Hallucination Detection in LLMs from Hidden Statescited by 2
Built on by
Not cited yet.