# VibeCheck: Discover and Quantify Qualitative Differences in Large Language Models

arXiv: https://arxiv.org/abs/2410.12851 · HTML: https://mlsanity.com/paper/2410.12851/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 10 Oct 2024.

## Built on by (most cited first, 17 of 17)

- [What Prompts Don't Say: Understanding and Managing Underspecification in LLM Prompts](https://mlsanity.com/paper/2505.13360/) (2025)
- [VisionArena: 230K Real World User-VLM Conversations with Preference Labels](https://mlsanity.com/paper/2412.08687/) (2024)
- [SciArena: An Open Evaluation Platform for Non-Verifiable Scientific Literature-Grounded Tasks](https://mlsanity.com/paper/2507.01001/) (2025)
- [EvalAgent: Discovering Implicit Evaluation Criteria from the Web](https://mlsanity.com/paper/2504.15219/) (2025)
- [How Well Does Agent Development Reflect Real-World Work?](https://mlsanity.com/paper/2603.01203/) (2026)
- [HyPerAlign: Interpretable Personalized LLM Alignment via Hypothesis Generation](https://mlsanity.com/paper/2505.00038/) (2025)
- [Simple LLM Baselines are Competitive for Model Diffing](https://mlsanity.com/paper/2602.10371/) (2026)
- [Helping Large Language Models Protect Themselves: An Enhanced Filtering and Summarization System](https://mlsanity.com/paper/2505.01315/) (2025)
- [Teaching Humans Subtle Differences with DIFFusion](https://mlsanity.com/paper/2504.08046/) (2025)
- [Generating Data-Driven Reasoning Rubrics for Domain-Adaptive Reward Modeling](https://mlsanity.com/paper/2602.06795/) (2026)
- [Asymmetric Idiosyncrasies in Multimodal Models](https://mlsanity.com/paper/2602.22734/) (2026)
- [Comparing Developer and LLM Biases in Code Evaluation](https://mlsanity.com/paper/2603.24586/) (2026)
- [Feedback Forensics: A Toolkit to Measure AI Personality](https://mlsanity.com/paper/2509.26305/) (2025)
- [From Feelings to Metrics: Understanding and Formalizing How Users Vibe-Test LLMs](https://mlsanity.com/paper/2604.14137/) (2026)
- [Language, Language Models, and What We're Talking About](https://mlsanity.com/paper/2609.03577/) (2026)
- [JudgeProfile: Understanding and Steering Subjectivity in LLM Judges](https://mlsanity.com/paper/2609.36705/) (2026)
- [Learning to Detect Shifts in Unstructured Data Streams via Monitoring-by-Betting](https://mlsanity.com/paper/2610.17399/) (2026)

## Abstract (arXiv)

Large language models (LLMs) often exhibit subtle yet distinctive characteristics in their outputs that users intuitively recognize, but struggle to quantify. These "vibes" -- such as tone, formatting, or writing style -- influence user preferences, yet traditional evaluations focus primarily on the singular axis of correctness. We introduce VibeCheck, a system for automatically comparing a pair of LLMs by discovering identifying traits of a model (vibes) that are well-defined, differentiating, and user-aligned. VibeCheck iteratively discovers vibes from model outputs and then utilizes a panel of LLM judges to quantitatively measure the utility of each vibe. We validate that the vibes generated by VibeCheck align with those found in human discovery and run VibeCheck on pairwise preference data from real-world user conversations with Llama-3-70b vs GPT-4. VibeCheck reveals that Llama has a friendly, funny, and somewhat controversial vibe. These vibes predict model identity with 80% accuracy and human preference with 61% accuracy. Lastly, we run VibeCheck on a variety of models and tasks including summarization, math, and captioning to provide insight into differences in model behavior. VibeCheck discovers vibes like Command X prefers to add concrete intros and conclusions when summarizing in comparison to TNGL, Llama-405b often overexplains its thought process on math problems compared to GPT-4o, and GPT-4 prefers to focus on the mood and emotions of the scene when captioning compared to Gemini-1.5-Flash. Code and vibe visualizer found at https://bench-mark.org/