# PEACH: Pretrained-embedding Explanation Across Contextual and Hierarchical Structure

arXiv: https://arxiv.org/abs/2404.13645 · HTML: https://mlsanity.com/paper/2404.13645/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 43 references that are in the map.
- First posted on arXiv 21 Apr 2024.

## Builds on (most cited first, 25 of 25)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [XLNet: Generalized Autoregressive Pretraining for Language Understanding](https://mlsanity.com/paper/1906.08237/) (2019)
- [ALBERT: A Lite BERT for Self-supervised Learning of Language Representations](https://mlsanity.com/paper/1909.11942/) (2019)
- [Deep contextualized word representations](https://mlsanity.com/paper/1802.05365/) (2018)
- [Program Induction by Rationale Generation : Learning to Solve and Explain Algebraic Word Problems](https://mlsanity.com/paper/1705.04146/) (2017)
- [Fooling LIME and SHAP: Adversarial Attacks on Post hoc Explanation Methods](https://mlsanity.com/paper/1911.02508/) (2019)
- [Is Attention Interpretable?](https://mlsanity.com/paper/1906.03731/) (2019)
- [Rationalizing Neural Predictions](https://mlsanity.com/paper/1606.04155/) (2016)
- [Neural Prototype Trees for Interpretable Fine-grained Image Recognition](https://mlsanity.com/paper/2012.02046/) (2020)
- [Representer Point Selection for Explaining Deep Neural Networks](https://mlsanity.com/paper/1811.09720/) (2018)
- [Explaining Black Box Predictions and Unveiling Data Artifacts through Influence Functions](https://mlsanity.com/paper/2005.06676/) (2020)
- [Learning to Deceive with Attention-Based Explanations](https://mlsanity.com/paper/1909.07913/) (2019)
- [CogTree: Cognition Tree Loss for Unbiased Scene Graph Generation](https://mlsanity.com/paper/2009.07526/) (2020)
- [SelfExplain: A Self-Explaining Architecture for Neural Text Classifiers](https://mlsanity.com/paper/2103.12279/) (2021)
- [Rationalization: A Neural Machine Translation Approach to Generating Natural Language Explanations](https://mlsanity.com/paper/1702.07826/) (2017)
- [Tree-structured Attention with Hierarchical Accumulation](https://mlsanity.com/paper/2002.08046/) (2020)
- [Learning from the Best: Rationalizing Prediction by Adversarial Information Calibration](https://mlsanity.com/paper/2012.08884/) (2020)
- [Towards Understanding Neural Machine Translation with Word Importance](https://mlsanity.com/paper/1909.00326/) (2019)
- [Learning from Explanations with Neural Execution Tree](https://mlsanity.com/paper/1911.01352/) (2019)
- [On Attribution of Recurrent Neural Network Predictions via Additive Decomposition](https://mlsanity.com/paper/1903.11245/) (2019)
- [Oblique Decision Trees from Derivatives of ReLU Networks](https://mlsanity.com/paper/1909.13488/) (2019)
- [Training Decision Trees as Replacement for Convolution Layers](https://mlsanity.com/paper/1905.10073/) (2019)
- [CDC: Classification Driven Compression for Bandwidth Efficient Edge-Cloud Collaborative Deep Learning](https://mlsanity.com/paper/2005.02177/) (2020)
- [Rationalizing Predictions by Adversarial Information Calibration](https://mlsanity.com/paper/2301.06009/) (2023)

## Built on by (most cited first, 1 of 1)

- [PaperFit: Vision-in-the-Loop Typesetting Optimization for Scientific Documents](https://mlsanity.com/paper/2605.10341/) (2026)

## Abstract (arXiv)

In this work, we propose a novel tree-based explanation technique, PEACH (Pretrained-embedding Explanation Across Contextual and Hierarchical Structure), that can explain how text-based documents are classified by using any pretrained contextual embeddings in a tree-based human-interpretable manner. Note that PEACH can adopt any contextual embeddings of the PLMs as a training input for the decision tree. Using the proposed PEACH, we perform a comprehensive analysis of several contextual embeddings on nine different NLP text classification benchmarks. This analysis demonstrates the flexibility of the model by applying several PLM contextual embeddings, its attribute selections, scaling, and clustering methods. Furthermore, we show the utility of explanations by visualising the feature selection and important trend of text classification via human-interpretable word-cloud-based trees, which clearly identify model mistakes and assist in dataset debugging. Besides interpretability, PEACH outperforms or is similar to those from pretrained models.