# Assessing Hidden Risks of LLMs: An Empirical Study on Robustness, Consistency, and Credibility

arXiv: https://arxiv.org/abs/2305.10235 · HTML: https://mlsanity.com/paper/2305.10235/

## Key facts

- Cited by 30 papers in the map (data through 14 Oct 2026).
- Builds on 49 of its 64 references that are in the map.
- First posted on arXiv 15 May 2023.

## Builds on (most cited first, 30 of 49)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [Towards Deep Learning Models Resistant to Adversarial Attacks](https://mlsanity.com/paper/1706.06083/) (2017)
- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [OPT: Open Pre-trained Transformer Language Models](https://mlsanity.com/paper/2205.01068/) (2022)
- [XLNet: Generalized Autoregressive Pretraining for Language Understanding](https://mlsanity.com/paper/1906.08237/) (2019)
- [Adversarial NLI: A New Benchmark for Natural Language Understanding](https://mlsanity.com/paper/1910.14599/) (2019)
- [Did Aristotle Use a Laptop? A Question Answering Benchmark with Implicit Reasoning Strategies](https://mlsanity.com/paper/2101.02235/) (2021)
- [Universal Adversarial Triggers for Attacking and Analyzing NLP](https://mlsanity.com/paper/1908.07125/) (2019)
- [Program Induction by Rationale Generation : Learning to Solve and Explain Algebraic Word Problems](https://mlsanity.com/paper/1705.04146/) (2017)
- [Adversarial Examples for Evaluating Reading Comprehension Systems](https://mlsanity.com/paper/1707.07328/) (2017)
- [BERT-ATTACK: Adversarial Attack Against BERT Using BERT](https://mlsanity.com/paper/2004.09984/) (2020)
- [Is ChatGPT a General-Purpose Natural Language Processing Task Solver?](https://mlsanity.com/paper/2302.06476/) (2023)
- [e-SNLI: Natural Language Inference with Natural Language Explanations](https://mlsanity.com/paper/1812.01193/) (2018)
- [Decision-Based Adversarial Attacks: Reliable Attacks Against Black-Box Machine Learning Models](https://mlsanity.com/paper/1712.04248/) (2017)
- [Generating Natural Language Adversarial Examples](https://mlsanity.com/paper/1804.07998/) (2018)
- [TextBugger: Generating Adversarial Text Against Real-world Applications](https://mlsanity.com/paper/1812.05271/) (2018)
- [Black-box Generation of Adversarial Text Sequences to Evade Deep Learning Classifiers](https://mlsanity.com/paper/1801.04354/) (2018)
- [BAE: BERT-based Adversarial Examples for Text Classification](https://mlsanity.com/paper/2004.01970/) (2020)
- [Synthetic and Natural Noise Both Break Neural Machine Translation](https://mlsanity.com/paper/1711.02173/) (2017)
- [Adversarial Example Generation with Syntactically Controlled Paraphrase Networks](https://mlsanity.com/paper/1804.06059/) (2018)
- [QASC: A Dataset for Question Answering via Sentence Composition](https://mlsanity.com/paper/1910.11473/) (2019)
- [Adversarial GLUE: A Multi-Task Benchmark for Robustness Evaluation of Language Models](https://mlsanity.com/paper/2111.02840/) (2021)
- [WARP: Word-level Adversarial ReProgramming](https://mlsanity.com/paper/2101.00121/) (2021)
- [On the Robustness of ChatGPT: An Adversarial and Out-of-distribution Perspective](https://mlsanity.com/paper/2302.12095/) (2023)
- [ChatGPT: Jack of all trades, master of none](https://mlsanity.com/paper/2302.10724/) (2023)
- [Generating Natural Adversarial Examples](https://mlsanity.com/paper/1710.11342/) (2017)
- [Is ChatGPT better than Human Annotators? Potential and Limitations of ChatGPT in Explaining Implicit Hate Speech](https://mlsanity.com/paper/2302.07736/) (2023)

## Built on by (most cited first, 30 of 30)

- [Can LLM-Generated Misinformation Be Detected?](https://mlsanity.com/paper/2309.13788/) (2023)
- [Safeguarding Large Language Models: A Survey](https://mlsanity.com/paper/2406.02622/) (2024)
- [GPTEval: A Survey on Assessments of ChatGPT and GPT-4](https://mlsanity.com/paper/2308.12488/) (2023)
- [TableGPT: Towards Unifying Tables, Nature Language and Commands into One GPT](https://mlsanity.com/paper/2307.08674/) (2023)
- [Hallucination Detection: Robustly Discerning Reliable Answers in Large Language Models](https://mlsanity.com/paper/2407.04121/) (2024)
- [Data Contamination Calibration for Black-box LLMs](https://mlsanity.com/paper/2405.11930/) (2024)
- [A Survey on Privacy Risks and Protection in Large Language Models](https://mlsanity.com/paper/2505.01976/) (2025)
- [NLPerturbator: Studying the Robustness of Code LLMs to Natural Language Variations](https://mlsanity.com/paper/2406.19783/) (2024)
- [Robust Prompt Optimization for Large Language Models Against Distribution Shifts](https://mlsanity.com/paper/2305.13954/) (2023)
- [The Order Effect: Investigating Prompt Sensitivity to Input Order in LLMs](https://mlsanity.com/paper/2502.04134/) (2025)
- [Prompt-Reverse Inconsistency: LLM Self-Inconsistency Beyond Generative Randomness and Prompt Paraphrasing](https://mlsanity.com/paper/2504.01282/) (2025)
- [AuditLLM: A Tool for Auditing Large Language Models Using Multiprobe Approach](https://mlsanity.com/paper/2402.09334/) (2024)
- [Assessing LLMs for Zero-shot Abstractive Summarization Through the Lens of Relevance Paraphrasing](https://mlsanity.com/paper/2406.03993/) (2024)
- [Towards More Trustworthy and Interpretable LLMs for Code through Syntax-Grounded Explanations](https://mlsanity.com/paper/2407.08983/) (2024)
- [Measuring Free-Form Decision-Making Inconsistency of Language Models in Military Crisis Simulations](https://mlsanity.com/paper/2410.13204/) (2024)
- [Evaluating Consistency and Reasoning Capabilities of Large Language Models](https://mlsanity.com/paper/2404.16478/) (2024)
- [Crossing Linguistic Horizons: Finetuning and Comprehensive Evaluation of Vietnamese Large Language Models](https://mlsanity.com/paper/2403.02715/) (2024)
- [Generalists vs. Specialists: Evaluating Large Language Models for Urdu](https://mlsanity.com/paper/2407.04459/) (2024)
- [DICE: A Framework for Dimensional and Contextual Evaluation of Language Models](https://mlsanity.com/paper/2504.10359/) (2025)
- [Do Language Models Think Consistently? A Study of Value Preferences Across Varying Response Lengths](https://mlsanity.com/paper/2506.02481/) (2025)
- [Linearly Decoding Refused Knowledge in Aligned Language Models](https://mlsanity.com/paper/2507.00239/) (2025)
- [Selecting Shots for Demographic Fairness in Few-Shot Learning with Large Language Models](https://mlsanity.com/paper/2311.08472/) (2023)
- [Prioritize Economy or Climate Action? Investigating ChatGPT Response Differences Based on Inferred Political Orientation](https://mlsanity.com/paper/2511.04706/) (2025)
- [Evaluating LLM Safety Under Repeated Inference via Accelerated Prompt Stress Testing](https://mlsanity.com/paper/2602.11786/) (2026)
- [Breaking the Silence: the Threats of Using LLMs in Software Engineering](https://mlsanity.com/paper/2312.08055/) (2023)
- [Waste Not, Want Not; Recycled Gumbel Noise Improves Consistency in Natural Language Generation](https://mlsanity.com/paper/2503.00831/) (2025)
- [Mapping the Trust Terrain: LLMs in Software Engineering -- Insights and Perspectives](https://mlsanity.com/paper/2503.13793/) (2025)
- [EPT Benchmark: Evaluation of Persian Trustworthiness in Large Language Models](https://mlsanity.com/paper/2509.06838/) (2025)
- [Pop Quiz Attack: Black-box Membership Inference Attacks Against Large Language Models](https://mlsanity.com/paper/2605.06423/) (2026)
- [Beyond Accuracy and Surface Fluency: Risk-Sensitive Evaluation of LLMs for Legal Clause Generation](https://mlsanity.com/paper/2609.22127/) (2026)

## Abstract (arXiv)

The recent popularity of large language models (LLMs) has brought a significant impact to boundless fields, particularly through their open-ended ecosystem such as the APIs, open-sourced models, and plugins. However, with their widespread deployment, there is a general lack of research that thoroughly discusses and analyzes the potential risks concealed. In that case, we intend to conduct a preliminary but pioneering study covering the robustness, consistency, and credibility of LLMs systems. With most of the related literature in the era of LLM uncharted, we propose an automated workflow that copes with an upscaled number of queries/responses. Overall, we conduct over a million queries to the mainstream LLMs including ChatGPT, LLaMA, and OPT. Core to our workflow consists of a data primitive, followed by an automated interpreter that evaluates these LLMs under different adversarial metrical systems. As a result, we draw several, and perhaps unfortunate, conclusions that are quite uncommon from this trendy community. Briefly, they are: (i)-the minor but inevitable error occurrence in the user-generated query input may, by chance, cause the LLM to respond unexpectedly; (ii)-LLMs possess poor consistency when processing semantically similar query input. In addition, as a side finding, we find that ChatGPT is still capable to yield the correct answer even when the input is polluted at an extreme level. While this phenomenon demonstrates the powerful memorization of the LLMs, it raises serious concerns about using such data for LLM-involved evaluation in academic development. To deal with it, we propose a novel index associated with a dataset that roughly decides the feasibility of using such data for LLM-involved evaluation. Extensive empirical studies are tagged to support the aforementioned claims.