mlsanityData to 14 Oct 2026

LMdiff: A Visual Diff Tool to Compare Language Models

arXiv 2111.01582 ↗First posted 2 Nov 2021cs.CL cs.HCcited by 17

Not on the map. Why?

Abstract (arXiv)

While different language models are ubiquitous in NLP, it is hard to contrast their outputs and identify which contexts one can handle better than the other. To address this question, we introduce LMdiff, a tool that visually compares probability distributions of two models that differ, e.g., through finetuning, distillation, or simply training with different parameter sizes. LMdiff allows the generation of hypotheses about model behavior by investigating text instances token by token and further assists in choosing these interesting text instances by identifying the most interesting phrases from large corpora. We showcase the applicability of LMdiff for hypothesis generation across multiple case studies. A demo is available at http://lmdiff.net .

Key facts. Cited by 17 papers in the map (data through 14 Oct 2026). Builds on 23 of its 32 references that are in the map. First posted on arXiv 2 Nov 2021.

Builds on

23 of 32 references.

References · 23

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  4. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,307
  5. 2020Extracting Training Data from Large Language Modelscited by 2,189
  6. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,822
  7. 2019BERT Rediscovers the Classical NLP Pipelinecited by 1,059
  8. 2019How Can We Know What Language Models Know?cited by 959
  9. 2020A Primer in BERTology: What we know about how BERT workscited by 815
  10. 2020XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalizationcited by 742
  11. 2018Gender Bias in Coreference Resolution: Evaluation and Debiasing Methodscited by 659
  12. 2019Defending Against Neural Fake Newscited by 559
11 more
  1. 2019GLTR: Statistical Detection and Visualization of Generated Textcited by 496
  2. 2019A Multiscale Visualization of Attention in the Transformer Modelcited by 292
  3. 2018Analysis Methods in Neural Language Processing: A Surveycited by 277
  4. 2021The GEM Benchmark: Natural Language Generation, its Evaluation and Metricscited by 239
  5. 2020Characterising Bias in Compressed Modelscited by 167
  6. 2020The Language Interpretability Tool: Extensible, Interactive Visualizations and Analysis for NLP Modelscited by 125
  7. 2019FairVis: Visual Analytics for Discovering Intersectional Bias in Machine Learningcited by 93
  8. 2019AllenNLP Interpret: A Framework for Explaining Predictions of NLP Modelscited by 74
  9. 2018Seq2Seq-Vis: A Visual Debugging Tool for Sequence-to-Sequence Modelscited by 69
  10. 2020jiant: A Software Toolkit for Research on General-Purpose Text Understanding Modelscited by 39
  11. 2018NeuroX: A Toolkit for Analyzing Individual Neurons in Neural Networkscited by 31

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 3; 2023: 3; 2024: 6; 2025: 1; 2026: 4.

17 papers.

Sort:

By area:

  1. 2022Interactive and Visual Prompt Engineering for Ad-hoc Task Adaptation with Large Language Modelscited by 70
  2. 2024LLM Comparator: Visual Analytics for Side-by-Side Evaluation of Large Language Modelscited by 29
  3. 2023Which Prompts Make The Difference? Data Prioritization For Efficient Human LLM Evaluationcited by 18
  4. 2022Visual Comparison of Language Model Adaptationcited by 14
  5. 2023R-U-SURE? Uncertainty-Aware Code Suggestions By Maximizing Utility Across Random User Intentscited by 10
  6. 2022Mediators: Conversational Agents Explaining NLP Model Behaviorcited by 7
  7. 2024KnowledgeVIS: Interpreting Language Models by Comparing Fill-in-the-Blank Promptscited by 7
  8. 2024Interactive Prompt Debugging with Sequence Saliencecited by 6
  9. 2023Visual Analytics for Generative Transformer Modelscited by 2
  10. 2026RAGExplorer: A Visual Analytics System for the Comparative Diagnosis of RAG Systemscited by 2
  11. 2024Consistency Matters: Explore LLMs Consistency From a Black-Box Perspectivecited by 1
  12. 2024SimCT: A Simple Consistency Test Protocol in LLMs Development Lifecyclecited by 1
5 more
  1. 2025ConceptViz: A Visual Analytics Approach for Exploring Concepts in Large Language Modelscited by 1
  2. 2026Lexara: A User-Centered Toolkit for Evaluating Large Language Models for Conversational Visual Analyticscited by 1
  3. 2024Model-diff: A Tool for Comparative Study of Language Models in the Input Spacecited by 0
  4. 2026Beyond One Output: Visualizing and Comparing Distributions of Language Model Generationscited by 0
  5. 2026Exposing the Unsaid: Visualizing Hidden LLM Bias through Stochastic Path Aggregationcited by 0