mlsanityData to 7 Oct 2026

LLM Comparator: Visual Analytics for Side-by-Side Evaluation of Large Language Models

arXiv 2402.10524 ↗First posted 16 Feb 2024cs.HC cs.AI cs.CLcited by 29

Not on the map. Why?

Abstract (arXiv)

Automatic side-by-side evaluation has emerged as a promising approach to evaluating the quality of responses from large language models (LLMs). However, analyzing the results from this evaluation approach raises scalability and interpretability challenges. In this paper, we present LLM Comparator, a novel visual analytics tool for interactively analyzing results from automatic side-by-side evaluation. The tool supports interactive workflows for users to understand when and why a model performs better or worse than a baseline model, and how the responses from two models are qualitatively different. We iteratively designed and developed the tool by closely working with researchers and engineers at a large technology company. This paper details the user challenges we identified, the design and development of the tool, and an observational study with participants who regularly evaluate their models.

Key facts. Cited by 29 papers in the map (data through 7 Oct 2026). Builds on 17 of its 25 references that are in the map. First posted on arXiv 16 Feb 2024.

Builds on

17 of 25 references.

References · 17

  1. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  2. 2016Concrete Problems in AI Safetycited by 1,582
  3. 2019Machine Learning Testing: Survey, Landscapes and Horizonscited by 213
  4. 2020The Language Interpretability Tool: Extensible, Interactive Visualizations and Analysis for NLP Modelscited by 125
  5. 2024LiPO: Listwise Preference Optimization through Learning-to-Rankcited by 94
  6. 2023ChainForge: A Visual Toolkit for Prompt Engineering and LLM Hypothesis Testingcited by 88
  7. 2023EvalLM: Interactive Evaluation of Large Language Model Prompts on User-Defined Criteriacited by 73
  8. 2022Interactive and Visual Prompt Engineering for Ad-hoc Task Adaptation with Large Language Modelscited by 70
  9. 2018RuleMatrix: Visualizing and Understanding Classifiers with Rulescited by 62
  10. 2023Goal-Driven Explainable Clustering via Language Descriptionscited by 58
  11. 2017ActiVis: Visual Exploration of Industry-Scale Deep Neural Network Modelscited by 56
  12. 2022Describing Differences between Text Distributions with Natural Languagecited by 54
5 more
  1. 2019Embedding Comparator: Visualizing Differences in Global Structure and Local Neighborhoods via Small Multiplescited by 38
  2. 2020Boxer: Interactive Comparison of Classifier Resultscited by 28
  3. 2021LMdiff: A Visual Diff Tool to Compare Language Modelscited by 17
  4. 2022Learning-From-Disagreement: A Model Comparison and Visual Analytics Frameworkcited by 9
  5. 2023The Role of Interactive Visualization in Explaining (Large) NLP Models: from Data to Inferencecited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 9; 2025: 10; 2026: 10.

29 papers.

Sort:

By area:

  1. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,928
  2. 2024A Toolbox for Surfacing Health Equity Harms and Biases in Large Language Modelscited by 45
  3. 2024Designing a Dashboard for Transparency and Control of Conversational AIcited by 43
  4. 2024Natural Language Outlines for Code: Literate Programming in the LLM Eracited by 9
  5. 2024Understanding Large Language Model Behaviors through Interactive Counterfactual Generation and Analysiscited by 8
  6. 2025DeepVIS: Bridging Natural Language and Data Visualization Through Step-wise Reasoningcited by 8
  7. 2024ChainBuddy: An AI Agent System for Generating LLM Pipelinescited by 6
  8. 2024Exploring Empty Spaces: Human-in-the-Loop Data Augmentationcited by 6
  9. 2025RAGTrace: Understanding and Refining Retrieval-Generation Dynamics in Retrieval-Augmented Generationcited by 4
  10. 2025DxHF: Providing High-Quality Human Feedback for LLM Alignment via Interactive Decompositioncited by 3
  11. 2025Illuminating LLM Coding Agents: Visual Analytics for Deeper Understanding and Enhancementcited by 3
  12. 2025Evalet: Evaluating Large Language Models through Functional Fragmentationcited by 3
17 more
  1. 2025ALIGN: Prompt-based Attribute Alignment for Reliable, Responsible, and Personalized LLM-based Decision-Makingcited by 2
  2. 2024JailbreakHunter: A Visual Analytics Approach for Jailbreak Prompts Discovery from Large-Scale Human-LLM Conversational Datasetscited by 1
  3. 2025Gensors: Authoring Personalized Visual Sensors with Multimodal Foundation Models and Reasoningcited by 1
  4. 2025Beyond Benchmarks: LLM Evaluation with an Anthropomorphic and Lifecycle-oriented Roadmapcited by 1
  5. 2026Understanding LLM Evaluator Behavior: A Structured Multi-Evaluator Framework for Merchant Risk Assessmentcited by 1
  6. 2026An Expert Schema for Evaluating Large Language Model Errors in Scholarly Question-Answering Systemscited by 1
  7. 2026Lexara: A User-Centered Toolkit for Evaluating Large Language Models for Conversational Visual Analyticscited by 1
  8. 2026Position: Multi-Agent Systems Should Prioritize Concurrency Controlcited by 1
  9. 2024MetaGreen: Meta-Learning Inspired Transformer Selection for Green Semantic Communicationcited by 0
  10. 2025A Deep User Interface for Exploring LLaMacited by 0
  11. 2025Uncovering Competency Gaps in Large Language Models and Their Benchmarkscited by 0
  12. 2026Zero-shot adaptable task planning for autonomous construction robots: a comparative study of lightweight single and multi-AI agent systemscited by 0
  13. 2026Discovering Implicit Large Language Model Alignment Objectivescited by 0
  14. 2026Beyond One Output: Visualizing and Comparing Distributions of Language Model Generationscited by 0
  15. 2026Visual Fingerprints for LLM Generation Comparisoncited by 0
  16. 2026Exposing the Unsaid: Visualizing Hidden LLM Bias through Stochastic Path Aggregationcited by 0
  17. 2026Token Signatures of Code: Comparing Coding Behaviors Across Large Language Modelscited by 0