mlsanityData to 14 Oct 2026

Measuring What LLMs Think They Do: SHAP Faithfulness and Deployability on Financial Tabular Classification

arXiv 2512.00163 ↗First posted 28 Nov 2025cs.LG cs.CLcited by 3

On the map: LLM reasoning → Reasoning for agents, planning and code

Abstract (arXiv)

Large Language Models (LLMs) have attracted significant attention for classification tasks, offering a flexible alternative to trusted classical machine learning models like LightGBM through zero-shot prompting. However, their reliability for structured tabular data remains unclear, particularly in high stakes applications like financial risk assessment. Our study systematically evaluates LLMs and generates their SHAP values on financial classification tasks. Our analysis shows a divergence between LLMs self-explanation of feature impact and their SHAP values, as well as notable differences between LLMs and LightGBM SHAP values. These findings highlight the limitations of LLMs as standalone classifiers for structured financial modeling, but also instill optimism that improved explainability mechanisms coupled with few-shot prompting will make LLMs usable in risk-sensitive domains.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Placed in Reasoning for agents, planning and code, one of 17 directions in LLM reasoning. Builds on 16 of its 33 references that are in the map. First posted on arXiv 28 Nov 2025.

Builds on

16 of 33 references.

Closest ancestors · 2

Same direction, newest first

  1. 2025A Survey of Frontiers in LLM Reasoning: Inference Scaling, Learning to Reason, and Agentic Systems12 Apr
  2. 2024Equipping Language Models with Tool Use Capability for Tabular Data Analysis in Finance27 Jan

Other references · 14

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2024The Llama 3 Herd of Modelscited by 14,715
  3. 2024Qwen2.5 Technical Reportcited by 4,942
  4. 2023Mistral 7Bcited by 3,164
  5. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,998
  6. 2017Towards A Rigorous Science of Interpretable Machine Learningcited by 1,416
  7. 2022Towards Reasoning in Large Language Models: A Surveycited by 596
  8. 2023Is ChatGPT a General-Purpose Natural Language Processing Task Solver?cited by 488
  9. 2023Can Large Language Models Explain Themselves? A Study of LLM-Generated Self-Explanationscited by 90
  10. 2024TokenSHAP: Interpreting Large Language Models with Monte Carlo Shapley Value Estimationcited by 42
  11. 2024Explaining Large Language Models Decisions Using Shapley Valuescited by 12
  12. 2024Large Language Models Cannot Explain Themselvescited by 11
2 more
  1. 2025Can LLMs Explain Themselves Counterfactually?cited by 10
  2. 2024Enhancing the Interpretability of SHAP Values Using Large Language Modelscited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026Signal or Noise in Multi-Agent LLM-based Stock Recommendations?cited by 1
  2. 2026LLM Doesn't Know What It Doesn't Know: Detecting Epistemic Blind Spots via Cross-Model Attribution Divergence on Clinical Tabular Datacited by 0
  3. 2026Operational Evidence Gaps for LLMs in Fraud Detection and Trust-and-Safety Workflowscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Interpreting LLMs as Credit Risk Classifiers: Do Their Feature Explanations Align with Classical ML?
  2. 2023Beyond Classification: Financial Reasoning in State-of-the-Art Language Models
  3. 2025Interpretable LLMs for Credit Risk: A Systematic Review and Taxonomy
  4. 2025Beyond the Black Box: Interpretability of LLMs in Finance
  5. 2025Understanding Structured Financial Data with LLMs: A Case Study on Fraud Detection
  6. 2026Are Large Language Models Reliable Reviewers? A Benchmark for Error Detection in Financial Documents
  7. 2025QuantMCP: Grounding Large Language Models in Verifiable Financial Reality
  8. 2025Evaluating Large Language Models (LLMs) in Financial NLP: A Comparative Study on Financial Report Analysis