# FINCH: Locally Visualizing Higher-Order Feature Interactions in Black Box Models

arXiv: https://arxiv.org/abs/2503.16445 · HTML: https://mlsanity.com/paper/2503.16445/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 32 references that are in the map.
- First posted on arXiv 17 Feb 2025.

## Builds on (most cited first, 14 of 14)

- ["Why Should I Trust You?": Explaining the Predictions of Any Classifier](https://mlsanity.com/paper/1602.04938/) (2016)
- [Explanation in Artificial Intelligence: Insights from the Social Sciences](https://mlsanity.com/paper/1706.07269/) (2017)
- [Consistent Individualized Feature Attribution for Tree Ensembles](https://mlsanity.com/paper/1802.03888/) (2018)
- [Visualizing the Effects of Predictor Variables in Black Box Supervised Learning Models](https://mlsanity.com/paper/1612.08468/) (2016)
- [A Survey of the State of Explainable AI for Natural Language Processing](https://mlsanity.com/paper/2010.00711/) (2020)
- [Model-agnostic Feature Importance and Effects with Dependent Features -- A Conditional Subgroup Approach](https://mlsanity.com/paper/2006.04628/) (2020)
- [REPID: Regional Effect Plots with implicit Interaction Detection](https://mlsanity.com/paper/2202.07254/) (2022)
- [Decomposing Global Feature Effects Based on Feature Interactions](https://mlsanity.com/paper/2306.00541/) (2023)
- [VINE: Visualizing Statistical Interactions in Black Box Models](https://mlsanity.com/paper/1904.00561/) (2019)
- [Human-interpretable model explainability on high-dimensional data](https://mlsanity.com/paper/2010.07384/) (2020)
- [Interpretable Artificial Intelligence through the Lens of Feature Interaction](https://mlsanity.com/paper/2103.03103/) (2021)
- [Why You Should Not Trust Interpretations in Machine Learning: Adversarial Attacks on Partial Dependence Plots](https://mlsanity.com/paper/2404.18702/) (2024)
- [Coalitional strategies for efficient individual prediction explanation](https://mlsanity.com/paper/2104.00765/) (2021)
- [Function Trees: Transparent Machine Learning](https://mlsanity.com/paper/2403.13141/) (2024)

## Built on by (most cited first, 2 of 2)

- [Interaction Tensor SHAP](https://mlsanity.com/paper/2512.05338/) (2025)
- [SAILS: Surrogate-based Analysis of Interactions via Local Effect Smooths](https://mlsanity.com/paper/2606.09404/) (2026)

## Abstract (arXiv)

In an era where black-box AI models are integral to decision-making across industries, robust methods for explaining these models are more critical than ever. While these models leverage complex feature interplay for accurate predictions, most explanation methods only assign relevance to individual features. There is a research gap in methods that effectively illustrate interactions between features, especially in visualizing higher-order interactions involving multiple features, which challenge conventional representation methods. To address this challenge in local explanations focused on individual instances, we employ a visual, subset-based approach to reveal relevant feature interactions. Our visual analytics tool FINCH uses coloring and highlighting techniques to create intuitive, human-centered visualizations, and provides additional views that enable users to calibrate their trust in the model and explanations. We demonstrate FINCH in multiple case studies, demonstrating its generalizability, and conducted an extensive human study with machine learning experts to highlight its helpfulness and usability. With this approach, FINCH allows users to visualize feature interactions involving any number of features locally.