# VERBA: Verbalizing Model Differences Using Large Language Models

arXiv: https://arxiv.org/abs/2507.02241 · HTML: https://mlsanity.com/paper/2507.02241/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 17 of its 31 references that are in the map.
- First posted on arXiv 3 Jul 2025.

## Builds on (most cited first, 17 of 17)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Model Cards for Model Reporting](https://mlsanity.com/paper/1810.03993/) (2018)
- [Eliciting Latent Predictions from Transformers with the Tuned Lens](https://mlsanity.com/paper/2303.08112/) (2023)
- [How Good Are GPT Models at Machine Translation? A Comprehensive Evaluation](https://mlsanity.com/paper/2302.09210/) (2023)
- [Patchscopes: A Unifying Framework for Inspecting Hidden Representations of Language Models](https://mlsanity.com/paper/2401.06102/) (2024)
- [Compositional Explanations of Neurons](https://mlsanity.com/paper/2006.14032/) (2020)
- [Data Cards: Purposeful and Transparent Dataset Documentation for Responsible AI](https://mlsanity.com/paper/2204.01075/) (2022)
- [Natural Language Descriptions of Deep Visual Features](https://mlsanity.com/paper/2201.11114/) (2022)
- [Future Lens: Anticipating Subsequent Tokens from a Single Hidden State](https://mlsanity.com/paper/2311.04897/) (2023)
- [Explaining black box text modules in natural language with language models](https://mlsanity.com/paper/2305.09863/) (2023)
- [Describing Differences between Text Distributions with Natural Language](https://mlsanity.com/paper/2201.12323/) (2022)
- [Rigorously Assessing Natural Language Explanations of Neurons](https://mlsanity.com/paper/2309.10312/) (2023)
- [Ranking and Tuning Pre-trained Models: A New Paradigm for Exploiting Model Hubs](https://mlsanity.com/paper/2110.10545/) (2021)
- [LLM-Generated Black-box Explanations Can Be Adversarially Helpful](https://mlsanity.com/paper/2405.06800/) (2024)
- [CoSy: Evaluating Textual Explanations of Neurons](https://mlsanity.com/paper/2405.20331/) (2024)
- [Automatic Generation of Model and Data Cards: A Step Towards Responsible AI](https://mlsanity.com/paper/2405.06258/) (2024)
- [Towards Generating Informative Textual Description for Neurons in Language Models](https://mlsanity.com/paper/2401.16731/) (2024)

## Abstract (arXiv)

In the current machine learning landscape, we face a "model lake" phenomenon: Given a task, there is a proliferation of trained models with similar performances despite different behavior. For model users attempting to navigate and select from the models, documentation comparing model pairs is helpful. However, for every N models there could be O(N2) pairwise comparisons, a number prohibitive for the model developers to manually perform pairwise comparisons and prepare documentations. To facilitate fine-grained pairwise comparisons among models, we introduced VERBA. Our approach leverages a large language model (LLM) to generate verbalizations of model differences by sampling from the two models. We established a protocol that evaluates the informativeness of the verbalizations via simulation. We also assembled a suite with a diverse set of commonly used machine learning models as a benchmark. For a pair of decision tree models with up to 5% performance difference but 20-25% behavioral differences, VERBA effectively verbalizes their variations with up to 80% overall accuracy. When we included the models' structural information, the verbalization's accuracy further improved to 90%. VERBA opens up new research avenues for improving the transparency and comparability of machine learning models in a post-hoc manner.