# Betti numbers of attention graphs is all you really need

arXiv: https://arxiv.org/abs/2207.01903 · HTML: https://mlsanity.com/paper/2207.01903/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 17 references that are in the map.
- First posted on arXiv 5 Jul 2022.

## Builds on (most cited first, 13 of 13)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [What Does BERT Look At? An Analysis of BERT's Attention](https://mlsanity.com/paper/1906.04341/) (2019)
- [Neural Network Acceptability Judgments](https://mlsanity.com/paper/1805.12471/) (2018)
- [Are Sixteen Heads Really Better than One?](https://mlsanity.com/paper/1905.10650/) (2019)
- [An introduction to Topological Data Analysis: fundamental and practical aspects for data scientists](https://mlsanity.com/paper/1710.04019/) (2017)
- [A roadmap for the computation of persistent homology](https://mlsanity.com/paper/1506.08903/) (2015)
- [Topology of deep neural networks](https://mlsanity.com/paper/2004.06093/) (2020)
- [Neural Persistence: A Complexity Measure for Deep Neural Networks Using Algebraic Topology](https://mlsanity.com/paper/1812.09764/) (2018)
- [Computing the Testing Error without a Testing Set](https://mlsanity.com/paper/2005.00450/) (2020)
- [Towards a topological-geometrical theory of group equivariant non-expansive operators for data analysis and machine learning](https://mlsanity.com/paper/1812.11832/) (2018)
- [Path homologies of deep feedforward networks](https://mlsanity.com/paper/1910.07617/) (2019)
- [Homotopy Theoretic and Categorical Models of Neural Information Networks](https://mlsanity.com/paper/2006.15136/) (2020)

## Built on by (most cited first, 1 of 1)

- [Characterization of topological structures in different neural network architectures](https://mlsanity.com/paper/2407.06286/) (2024)

## Abstract (arXiv)

We apply methods of topological analysis to the attention graphs, calculated on the attention heads of the BERT model ( arXiv:1810.04805v2 ). Our research shows that the classifier built upon basic persistent topological features (namely, Betti numbers) of the trained neural network can achieve classification results on par with the conventional classification method. We show the relevance of such topological text representation on three text classification benchmarks. For the best of our knowledge, it is the first attempt to analyze the topology of an attention-based neural network, widely used for Natural Language Processing.