# Models in the Same Family are NOT Trust-Equivalent

arXiv: https://arxiv.org/abs/2508.13533 · HTML: https://mlsanity.com/paper/2508.13533/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 22 of its 30 references that are in the map.
- First posted on arXiv 19 Aug 2025.

## Builds on (most cited first, 22 of 22)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [A Unified Approach to Interpreting Model Predictions](https://mlsanity.com/paper/1705.07874/) (2017)
- ["Why Should I Trust You?": Explaining the Predictions of Any Classifier](https://mlsanity.com/paper/1602.04938/) (2016)
- [On Calibration of Modern Neural Networks](https://mlsanity.com/paper/1706.04599/) (2017)
- [OPT: Open Pre-trained Transformer Language Models](https://mlsanity.com/paper/2205.01068/) (2022)
- [QLoRA: Efficient Finetuning of Quantized LLMs](https://mlsanity.com/paper/2305.14314/) (2023)
- [A large annotated corpus for learning natural language inference](https://mlsanity.com/paper/1508.05326/) (2015)
- [Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling](https://mlsanity.com/paper/2304.01373/) (2023)
- [Concrete Problems in AI Safety](https://mlsanity.com/paper/1606.06565/) (2016)
- [Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddings](https://mlsanity.com/paper/1607.06520/) (2016)
- [A Survey Of Methods For Explaining Black Box Models](https://mlsanity.com/paper/1802.01933/) (2018)
- [DecodingTrust: A Comprehensive Assessment of Trustworthiness in GPT Models](https://mlsanity.com/paper/2306.11698/) (2023)
- [CheXNet: Radiologist-Level Pneumonia Detection on Chest X-Rays with Deep Learning](https://mlsanity.com/paper/1711.05225/) (2017)
- [On the Robustness of Interpretability Methods](https://mlsanity.com/paper/1806.08049/) (2018)
- [What Do Compressed Deep Neural Networks Forget?](https://mlsanity.com/paper/1911.05248/) (2019)
- [Evaluating model calibration in classification](https://mlsanity.com/paper/1902.06977/) (2019)
- [Adversarial Robustness vs Model Compression, or Both?](https://mlsanity.com/paper/1903.12561/) (2019)
- [Compression of Deep Learning Models for Text: A Survey](https://mlsanity.com/paper/2008.05221/) (2020)
- [Decoding Compressed Trust: Scrutinizing the Trustworthiness of Efficient LLMs Under Compression](https://mlsanity.com/paper/2403.15447/) (2024)
- [Membership Inference Attacks and Defenses in Neural Network Pruning](https://mlsanity.com/paper/2202.03335/) (2022)
- [Compressed models are NOT miniature versions of large models](https://mlsanity.com/paper/2407.13174/) (2024)

## Abstract (arXiv)

Within a model family, a smaller variant is often deployed as a drop-in replacement for a larger one when their performance is similar. However, performance alone does not tell the full story. We propose a framework to evaluate trust-equivalence between a larger model and a smaller one in the same family along two dimensions. The first is attribution alignment: do both models base their predictions on the same input features? The second is calibration similarity: do both models share the same relationship between confidence and accuracy? We evaluate the Llama-2 family on two text classification tasks: Natural Language Inference and Paraphrase Identification. Attribution alignment is measured using two well-known methods: LIME and SHAP. Agreement between model pairs is quantified via the Jaccard coefficient over top-K attributed features. We observe that attribution alignment between models is generally low, indicating that smaller and larger models base their predictions on different input features. Calibration similarity is assessed using ECE, MCE, Brier Score, and Reliability Diagrams. Calibration profiles differ substantially across model sizes. There is no consistent relationship between model size and calibration quality. We have additionally verified these trends on two encoder-only families: BERT and Vision Transformer. The results are consistent with those reported here. Our experimental results show that replacing a larger model with a smaller one from the same family is a multidimensional decision that requires consideration beyond performance measures alone. Trust-equivalence must be assessed explicitly. It cannot be assumed from performance alone.