# Topological Interpretations of GPT-3

arXiv: https://arxiv.org/abs/2308.03565 · HTML: https://mlsanity.com/paper/2308.03565/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 10 of its 21 references that are in the map.
- First posted on arXiv 7 Aug 2023.

## Builds on (most cited first, 10 of 10)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [Efficient Estimation of Word Representations in Vector Space](https://mlsanity.com/paper/1301.3781/) (2013)
- [Topology of deep neural networks](https://mlsanity.com/paper/2004.06093/) (2020)
- [Geometry Helps to Compare Persistence Diagrams](https://mlsanity.com/paper/1606.03357/) (2016)
- [Measuring Emotions in the COVID-19 Real World Worry Dataset](https://mlsanity.com/paper/2004.04225/) (2020)
- [Greedy Matroid Algorithm And Computational Persistent Homology](https://mlsanity.com/paper/2308.01796/) (2023)

## Built on by (most cited first, 2 of 2)

- [Topological Data Analysis Applications in Natural Language Processing: A Survey](https://mlsanity.com/paper/2411.10298/) (2024)
- [Vision Transformer Pruning Via Matrix Decomposition](https://mlsanity.com/paper/2308.10839/) (2023)

## Abstract (arXiv)

This is an experiential study of investigating a consistent method for deriving the correlation between sentence vector and semantic meaning of a sentence. We first used three state-of-the-art word/sentence embedding methods including GPT-3, Word2Vec, and Sentence-BERT, to embed plain text sentence strings into high dimensional spaces. Then we compute the pairwise distance between any possible combination of two sentence vectors in an embedding space and map them into a matrix. Based on each distance matrix, we compute the correlation of distances of a sentence vector with respect to the other sentence vectors in an embedding space. Then we compute the correlation of each pair of the distance matrices. We observed correlations of the same sentence in different embedding spaces and correlations of different sentences in the same embedding space. These observations are consistent with our hypothesis and take us to the next stage.