# Uncovering the Fragility of Trustworthy LLMs through Chinese Textual Ambiguity

arXiv: https://arxiv.org/abs/2507.23121 · HTML: https://mlsanity.com/paper/2507.23121/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 54 references that are in the map.
- First posted on arXiv 30 Jul 2025.

## Builds on (most cited first, 25 of 25)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [GPQA: A Graduate-Level Google-Proof Q&A Benchmark](https://mlsanity.com/paper/2311.12022/) (2023)
- [XLNet: Generalized Autoregressive Pretraining for Language Understanding](https://mlsanity.com/paper/1906.08237/) (2019)
- [Gemma 2: Improving Open Language Models at a Practical Size](https://mlsanity.com/paper/2408.00118/) (2024)
- [MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark](https://mlsanity.com/paper/2406.01574/) (2024)
- [DetectGPT: Zero-Shot Machine-Generated Text Detection using Probability Curvature](https://mlsanity.com/paper/2301.11305/) (2023)
- [A Survey on Large Language Models for Recommendation](https://mlsanity.com/paper/2305.19860/) (2023)
- [ERNIE: Enhanced Representation through Knowledge Integration](https://mlsanity.com/paper/1904.09223/) (2019)
- [Revisiting Pre-Trained Models for Chinese Natural Language Processing](https://mlsanity.com/paper/2004.13922/) (2020)
- [Large Language Models in Finance: A Survey](https://mlsanity.com/paper/2311.10723/) (2023)
- [A Survey on Uncertainty Quantification of Large Language Models: Taxonomy, Open Research Challenges, and Future Directions](https://mlsanity.com/paper/2412.05563/) (2024)
- [A Survey on Uncertainty Quantification Methods for Deep Learning](https://mlsanity.com/paper/2302.13425/) (2023)
- [Towards Few-Shot Fact-Checking via Perplexity](https://mlsanity.com/paper/2103.09535/) (2021)
- [Unifying the Perspectives of NLP and Software Engineering: A Survey on Language Models for Code](https://mlsanity.com/paper/2311.07989/) (2023)
- [CPM: A Large-scale Generative Chinese Pre-trained Language Model](https://mlsanity.com/paper/2012.00413/) (2020)
- [SKEP: Sentiment Knowledge Enhanced Pre-training for Sentiment Analysis](https://mlsanity.com/paper/2005.05635/) (2020)
- [Current state of LLM Risks and AI Guardrails](https://mlsanity.com/paper/2406.12934/) (2024)
- [Unmasking Digital Falsehoods: A Comparative Analysis of LLM-Based Misinformation Detection Strategies](https://mlsanity.com/paper/2503.00724/) (2025)
- [Do LLMs Understand Ambiguity in Text? A Case Study in Open-world Question Answering](https://mlsanity.com/paper/2411.12395/) (2024)
- [From None to Severe: Predicting Severity in Movie Scripts](https://mlsanity.com/paper/2109.09276/) (2021)
- [Positive and Risky Message Assessment for Music Products](https://mlsanity.com/paper/2309.10182/) (2023)
- [Birds of a Feather Flock Together: Satirical News Detection via Language Model Differentiation](https://mlsanity.com/paper/2007.02164/) (2020)
- [A Hybrid Transformer Model for Fake News Detection: Leveraging Bayesian Optimization and Bidirectional Recurrent Unit](https://mlsanity.com/paper/2502.09097/) (2025)

## Built on by (most cited first, 4 of 4)

- [Aligning Perception, Reasoning, Modeling and Interaction: A Survey on Physical AI](https://mlsanity.com/paper/2510.04978/) (2025)
- [EmbeddingRWKV: State-Centric Retrieval with Reusable States](https://mlsanity.com/paper/2601.07861/) (2026)
- [Evaluating Chinese Ambiguity Understanding in Large Language Models](https://mlsanity.com/paper/2605.15635/) (2026)
- [Tracing the ongoing emergence of human-like reasoning in Large Language Models](https://mlsanity.com/paper/2605.21299/) (2026)

## Abstract (arXiv)

In this work, we study a critical research problem regarding the trustworthiness of large language models (LLMs): how LLMs behave when encountering ambiguous narrative text, with a particular focus on Chinese textual ambiguity. We created a benchmark dataset by collecting and generating ambiguous sentences with context and their corresponding disambiguated pairs, representing multiple possible interpretations. These annotated examples are systematically categorized into 3 main categories and 9 subcategories. Through experiments, we discovered significant fragility in LLMs when handling ambiguity, revealing behavior that differs substantially from humans. Specifically, LLMs cannot reliably distinguish ambiguous text from unambiguous text, show overconfidence in interpreting ambiguous text as having a single meaning rather than multiple meanings, and exhibit overthinking when attempting to understand the various possible meanings. Our findings highlight a fundamental limitation in current LLMs that has significant implications for their deployment in real-world applications where linguistic ambiguity is common, calling for improved approaches to handle uncertainty in language understanding. The dataset and code are publicly available at this GitHub repository: https://github.com/ictup/LLM-Chinese-Textual-Disambiguation.