# Measuring What LLMs Think They Do: SHAP Faithfulness and Deployability on Financial Tabular Classification

arXiv: https://arxiv.org/abs/2512.00163 · HTML: https://mlsanity.com/paper/2512.00163/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Placed in Reasoning for agents, planning and code, one of 17 directions in LLM reasoning.
- Builds on 16 of its 33 references that are in the map.
- First posted on arXiv 28 Nov 2025.

## Placed in

- LLM reasoning / [Reasoning for agents, planning and code](https://mlsanity.com/lr/reasoning-for-agents-planning-and-code/)

## Builds on (most cited first, 16 of 16)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Qwen2.5 Technical Report](https://mlsanity.com/paper/2412.15115/) (2024)
- [Mistral 7B](https://mlsanity.com/paper/2310.06825/) (2023)
- [Gemma 2: Improving Open Language Models at a Practical Size](https://mlsanity.com/paper/2408.00118/) (2024)
- [Towards A Rigorous Science of Interpretable Machine Learning](https://mlsanity.com/paper/1702.08608/) (2017)
- [Towards Reasoning in Large Language Models: A Survey](https://mlsanity.com/paper/2212.10403/) (2022)
- [Is ChatGPT a General-Purpose Natural Language Processing Task Solver?](https://mlsanity.com/paper/2302.06476/) (2023)
- [A Survey of Frontiers in LLM Reasoning: Inference Scaling, Learning to Reason, and Agentic Systems](https://mlsanity.com/paper/2504.09037/) (2025)
- [Can Large Language Models Explain Themselves? A Study of LLM-Generated Self-Explanations](https://mlsanity.com/paper/2310.11207/) (2023)
- [TokenSHAP: Interpreting Large Language Models with Monte Carlo Shapley Value Estimation](https://mlsanity.com/paper/2407.10114/) (2024)
- [Equipping Language Models with Tool Use Capability for Tabular Data Analysis in Finance](https://mlsanity.com/paper/2401.15328/) (2024)
- [Explaining Large Language Models Decisions Using Shapley Values](https://mlsanity.com/paper/2404.01332/) (2024)
- [Large Language Models Cannot Explain Themselves](https://mlsanity.com/paper/2405.04382/) (2024)
- [Can LLMs Explain Themselves Counterfactually?](https://mlsanity.com/paper/2502.18156/) (2025)
- [Enhancing the Interpretability of SHAP Values Using Large Language Models](https://mlsanity.com/paper/2409.00079/) (2024)

## Built on by (most cited first, 3 of 3)

- [Signal or Noise in Multi-Agent LLM-based Stock Recommendations?](https://mlsanity.com/paper/2604.17327/) (2026)
- [LLM Doesn't Know What It Doesn't Know: Detecting Epistemic Blind Spots via Cross-Model Attribution Divergence on Clinical Tabular Data](https://mlsanity.com/paper/2606.19509/) (2026)
- [Operational Evidence Gaps for LLMs in Fraud Detection and Trust-and-Safety Workflows](https://mlsanity.com/paper/2607.13078/) (2026)

## Abstract (arXiv)

Large Language Models (LLMs) have attracted significant attention for classification tasks, offering a flexible alternative to trusted classical machine learning models like LightGBM through zero-shot prompting. However, their reliability for structured tabular data remains unclear, particularly in high stakes applications like financial risk assessment. Our study systematically evaluates LLMs and generates their SHAP values on financial classification tasks. Our analysis shows a divergence between LLMs self-explanation of feature impact and their SHAP values, as well as notable differences between LLMs and LightGBM SHAP values. These findings highlight the limitations of LLMs as standalone classifiers for structured financial modeling, but also instill optimism that improved explainability mechanisms coupled with few-shot prompting will make LLMs usable in risk-sensitive domains.