# Perplexed: Understanding When Large Language Models are Confused

arXiv: https://arxiv.org/abs/2404.06634 · HTML: https://mlsanity.com/paper/2404.06634/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 9 Apr 2024.

## Built on by (most cited first, 9 of 9)

- [Stepwise Perplexity-Guided Refinement for Efficient Chain-of-Thought Reasoning in Large Language Models](https://mlsanity.com/paper/2502.13260/) (2025)
- [EfficientPosterGen: Semantic-aware Efficient Poster Generation via Token Compression and Accurate Violation Detection](https://mlsanity.com/paper/2603.00155/) (2026)
- [Do LVLMs Know What They Know? A Systematic Study of Knowledge Boundary Perception in LVLMs](https://mlsanity.com/paper/2508.19111/) (2025)
- [Rethinking Code Complexity Through the Lens of Large Language Models](https://mlsanity.com/paper/2602.07882/) (2026)
- [Code Is More Than Text: Uncertainty Estimation for Code Generation](https://mlsanity.com/paper/2606.09577/) (2026)
- [Prompts Generalize with Low Data: Non-vacuous Generalization Bounds for Optimizing Prompts with More Informative Priors](https://mlsanity.com/paper/2510.08413/) (2025)
- [XGen-Q: An Explainable Domain-Adaptive LLM Framework with Retrieval-Augmented Generation for Software Security](https://mlsanity.com/paper/2510.19006/) (2025)
- [Is my model perplexed for the right reason? Contrasting LLMs' Benchmark Behavior with Token-Level Perplexity](https://mlsanity.com/paper/2603.29396/) (2026)
- [Understanding the Performance Plateau in Text-to-Video Retrieval: A Comprehensive Empirical and Linguistic Analysis](https://mlsanity.com/paper/2605.00826/) (2026)

## Abstract (arXiv)

Large Language Models (LLMs) have become dominant in the Natural Language Processing (NLP) field causing a huge surge in progress in a short amount of time. However, their limitations are still a mystery and have primarily been explored through tailored datasets to analyze a specific human-level skill such as negation, name resolution, etc. In this paper, we introduce perplexed, a library for exploring where a particular language model is perplexed. To show the flexibility and types of insights that can be gained by perplexed, we conducted a case study focused on LLMs for code generation using an additional tool we built to help with the analysis of code models called codetokenizer. Specifically, we explore success and failure cases at the token level of code LLMs under different scenarios pertaining to the type of coding structure the model is predicting, e.g., a variable name or operator, and how predicting of internal verses external method invocations impact performance. From this analysis, we found that our studied code LLMs had their worst performance on coding structures where the code was not syntactically correct. Additionally, we found the models to generally perform worse at predicting internal method invocations than external ones. We have open sourced both of these tools to allow the research community to better understand LLMs in general and LLMs for code generation.