# Visual Fingerprints for LLM Generation Comparison

arXiv: https://arxiv.org/abs/2605.06054 · HTML: https://mlsanity.com/paper/2605.06054/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 24 of its 38 references that are in the map.
- First posted on arXiv 7 May 2026.

## Builds on (most cited first, 24 of 24)

- [MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark](https://mlsanity.com/paper/2406.01574/) (2024)
- [BERTopic: Neural topic modeling with a class-based TF-IDF procedure](https://mlsanity.com/paper/2203.05794/) (2022)
- [Judging the Judges: Evaluating Alignment and Vulnerabilities in LLMs-as-Judges](https://mlsanity.com/paper/2406.12624/) (2024)
- [Bias Runs Deep: Implicit Reasoning Biases in Persona-Assigned LLMs](https://mlsanity.com/paper/2311.04892/) (2023)
- [Language Model Behavior: A Comprehensive Survey](https://mlsanity.com/paper/2303.11504/) (2023)
- [ChainForge: A Visual Toolkit for Prompt Engineering and LLM Hypothesis Testing](https://mlsanity.com/paper/2309.09128/) (2023)
- [EvalLM: Interactive Evaluation of Large Language Model Prompts on User-Defined Criteria](https://mlsanity.com/paper/2309.13633/) (2023)
- [Do Language Models Plagiarize?](https://mlsanity.com/paper/2203.07618/) (2022)
- [Graphologue: Exploring Large Language Model Responses with Interactive Diagrams](https://mlsanity.com/paper/2305.11473/) (2023)
- [Should We Respect LLMs? A Cross-Lingual Study on the Influence of Prompt Politeness on LLM Performance](https://mlsanity.com/paper/2402.14531/) (2024)
- [LLM Comparator: Visual Analytics for Side-by-Side Evaluation of Large Language Models](https://mlsanity.com/paper/2402.10524/) (2024)
- [Experimental Narratives: A Comparison of Human Crowdsourced Storytelling and AI Storytelling](https://mlsanity.com/paper/2310.12902/) (2023)
- [Supporting Sensemaking of Large Language Model Outputs at Scale](https://mlsanity.com/paper/2401.13726/) (2024)
- [RELIC: Investigating Large Language Model Responses using Self-Consistency](https://mlsanity.com/paper/2311.16842/) (2023)
- [Scattertext: a Browser-Based Tool for Visualizing how Corpora Differ](https://mlsanity.com/paper/1703.00565/) (2017)
- [Interactive Reasoning: Visualizing and Controlling Chain-of-Thought Reasoning in Large Language Models](https://mlsanity.com/paper/2506.23678/) (2025)
- [From Benchmarks to Skills: Low-Rank Factors for LLM Evaluation](https://mlsanity.com/paper/2507.20208/) (2025)
- [Automatic Histograms: Leveraging Language Models for Text Dataset Exploration](https://mlsanity.com/paper/2402.14880/) (2024)
- [Beyond the Singular: Revealing the Value of Multiple Generations in Benchmark Evaluation](https://mlsanity.com/paper/2502.08943/) (2025)
- [DxHF: Providing High-Quality Human Feedback for LLM Alignment via Interactive Decomposition](https://mlsanity.com/paper/2507.18802/) (2025)
- [Evalet: Evaluating Large Language Models through Functional Fragmentation](https://mlsanity.com/paper/2509.11206/) (2025)
- [Evaluating Ill-Defined Tasks in Large Language Models](https://mlsanity.com/paper/2603.17067/) (2026)
- [Word Storms: Multiples of Word Clouds for Visual Comparison of Documents](https://mlsanity.com/paper/1301.0503/) (2013)
- [Using language models to label clusters of scientific documents](https://mlsanity.com/paper/2511.02601/) (2025)

## Abstract (arXiv)

Large language model (LLM) outputs arise from complex interactions among prompts, system instructions, model parameters, and architecture. We refer to specific configurations of these factors as generation conditions, each of which can bias outputs in various ways. Understanding how different generation conditions shape model behaviors is essential for tasks such as prompt design and model evaluation, yet it remains challenging due to the stochastic and open-ended nature of text generation. We present an approach to visually compare LLM outputs across generation conditions by modeling responses as collections of linguistic choices, including content, expression, and structure. We extract these choices using natural language processing pipelines and represent their distributions across repeated samples. We then visualize these distributions as visual fingerprints, enabling direct, distribution-level comparison of condition-specific tendencies. Through four usage scenarios, we demonstrate how visual fingerprints reveal consistent patterns in LLM behavior that are difficult to observe through individual responses or aggregate metrics.