Harmonic LLMs are Trustworthy
Not on the map. Why?
Abstract (arXiv)
We introduce an intuitive method to test the robustness (stability and explainability) of any black-box LLM in real-time via its local deviation from harmoniticity, denoted as γ. To the best of our knowledge this is the first completely model-agnostic and unsupervised method of measuring the robustness of any given response from an LLM, based upon the model itself conforming to a purely mathematical standard. To show general application and immediacy of results, we measure γ in 10 popular LLMs (ChatGPT, Claude-2.1, Claude3.0, GPT-4, GPT-4o, Smaug-72B, Mixtral-8x7B, Llama2-7B, Mistral-7B and MPT-7B) across thousands of queries in three objective domains: WebQA, ProgrammingQA, and TruthfulQA. Across all models and domains tested, human annotation confirms that γ → 0 indicates trustworthiness, and conversely searching higher values of γ easily exposes examples of hallucination, a fact that enables efficient adversarial prompt generation through stochastic gradient ascent in γ. The low-γ leaders among the models in the respective domains are GPT-4o, GPT-4, and Smaug-72B, providing evidence that mid-size open-source models can win out against large commercial models.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 20 of its 24 references that are in the map. First posted on arXiv 30 Apr 2024.
Builds on
20 of 24 references.
References · 20
- 2024Mixtral of Expertscited by 1,823
- 2023Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMscited by 974
- 2023SelfCheckGPT: Zero-Resource Black-Box Hallucination Detection for Generative Large Language Modelscited by 810
- 2017Adversarial Examples for Evaluating Reading Comprehension Systemscited by 571
- 2023PromptRobust: Towards Evaluating the Robustness of Large Language Models on Adversarial Promptscited by 226
- 2023FLASK: Fine-grained Language Model Evaluation based on Alignment Skill Setscited by 201
- 2023Self-contradictory Hallucinations of Large Language Models: Evaluation, Detection and Mitigationcited by 124
- 2023Do Language Models Know When They're Hallucinating References?cited by 116
- 2023Flocks of Stochastic Parrots: Differentially Private Prompt Learning for Large Language Modelscited by 83
- 2020A Large-Scale Chinese Short-Text Conversation Datasetcited by 77
- 2023Attack Prompt Generation for Red Teaming and Defending Large Language Modelscited by 76
- 2019Perturbation Sensitivity Analysis to Detect Unintended Model Biasescited by 74
8 more
- 2023Evaluating the Zero-shot Robustness of Instruction-tuned Language Modelscited by 66
- 2023Are Large Language Models Really Robust to Word-Level Perturbations?cited by 32
- 2023Certified Robustness for Large Language Models with Self-Denoisingcited by 23
- 2023Revisit Input Perturbation Problems for LLMs: A Unified Robustness Evaluation Framework for Noisy Slot Filling Taskcited by 23
- 2021Interpreting the Robustness of Neural NLP Models to Textual Perturbationscited by 17
- 2023Methods to Estimate Large Language Model Confidencecited by 4
- 2023Exploring the Robustness of Model-Graded Evaluations and Automated Interpretabilitycited by 2
- 2024Harmonic Machine Learning Models are Robustcited by 1
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2025AdversariaLLM: A Unified and Modular Toolbox for LLM Robustness Research
- 2026DECK: A Consistency x Confidence Taxonomy of LLM Hallucinations
- 2025Auditing Black-Box LLM APIs with a Rank-Based Uniformity Test
- 2026TrustDABench: Benchmarking Reliability and Robustness of LLMs for Structured Data Analysis
- 2026All Verdicts are Not Equal: Rethinking LLM Judge Reliability
- 2025Can You Trust an LLM with Your Life-Changing Decision? An Investigation into AI High-Stakes Responses
- 2026Safeguarding LLMs via Model-Agnostic Latent Safety Signals from Dark Knowledge
- 2024LLMs Know More Than They Show: On the Intrinsic Representation of LLM Hallucinations