# Symbolic Rule Extraction from Attention-Guided Sparse Representations in Vision Transformers

arXiv: https://arxiv.org/abs/2505.06745 · HTML: https://mlsanity.com/paper/2505.06745/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 29 references that are in the map.
- First posted on arXiv 10 May 2025.

## Builds on (most cited first, 13 of 13)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Training data-efficient image transformers & distillation through attention](https://mlsanity.com/paper/2012.12877/) (2020)
- [Supervised Contrastive Learning](https://mlsanity.com/paper/2004.11362/) (2020)
- [Sparse autoencoders reveal selective remapping of visual concepts during adaptation](https://mlsanity.com/paper/2412.05276/) (2024)
- [Constraint Answer Set Programming without Grounding](https://mlsanity.com/paper/1804.11162/) (2018)
- [Steering CLIP's vision transformer with sparse autoencoders](https://mlsanity.com/paper/2504.08729/) (2025)
- [LeGrad: An Explainability Method for Vision Transformers via Feature Formation Sensitivity](https://mlsanity.com/paper/2404.03214/) (2024)
- [FOLD-SE: An Efficient Rule-based Machine Learning Algorithm with Scalable Explainability](https://mlsanity.com/paper/2208.07912/) (2022)
- [A Simple Interpretable Transformer for Fine-Grained Image Classification and Analysis](https://mlsanity.com/paper/2311.04157/) (2023)
- [ERIC: Extracting Relations Inferred from Convolutions](https://mlsanity.com/paper/2010.09452/) (2020)
- [Using Logic Programming and Kernel-Grouping for Improving Interpretability of Convolutional Neural Networks](https://mlsanity.com/paper/2310.13073/) (2023)

## Built on by (most cited first, 3 of 3)

- [Visual Commonsense Driven Knowledge Refinements for Scene Graph Generation](https://mlsanity.com/paper/2606.06369/) (2026)
- [Answer Set Programming Energised! End-to-End Neurosymbolic Reasoning and Learning with ASP and Energy Based Models](https://mlsanity.com/paper/2607.08136/) (2026)
- [SpIn-ViT: Designing a Sparsity-Induced Vision Transformer That Is Mechanistically Interpretable](https://mlsanity.com/paper/2608.14922/) (2026)

## Abstract (arXiv)

Recent neuro-symbolic approaches have successfully extracted symbolic rule-sets from CNN-based models to enhance interpretability. However, applying similar techniques to Vision Transformers (ViTs) remains challenging due to their lack of modular concept detectors and reliance on global self-attention mechanisms. We propose a framework for symbolic rule extraction from ViTs by introducing a sparse concept layer inspired by Sparse Autoencoders (SAEs). This linear layer operates on attention-weighted patch representations and learns a disentangled, binarized representation in which individual neurons activate for high-level visual concepts. To encourage interpretability, we apply a combination of L1 sparsity, entropy minimization, and supervised contrastive loss. These binarized concept activations are used as input to the FOLD-SE-M algorithm, which generates a rule-set in the form of logic programs. Our method achieves a 5.14% better classification accuracy than the standard ViT while enabling symbolic reasoning. Crucially, the extracted rule-set is not merely post-hoc but acts as a logic-based decision layer that operates directly on the sparse concept representations. The resulting programs are concise and semantically meaningful. This work is the first to extract executable logic programs from ViTs using sparse symbolic representations. It bridges the gap between transformer-based vision models and symbolic logic programming, providing a step forward in interpretable and verifiable neuro-symbolic AI.