# Beyond Accuracy: Measuring Representation Capacity of Embeddings to Preserve Structural and Contextual Information

arXiv: https://arxiv.org/abs/2309.11294 · HTML: https://mlsanity.com/paper/2309.11294/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 18 of its 46 references that are in the map.
- First posted on arXiv 20 Sep 2023.

## Builds on (most cited first, 18 of 18)

- [Unsupervised Deep Embedding for Clustering Analysis](https://mlsanity.com/paper/1511.06335/) (2015)
- [Tabular Data: Deep Learning is Not All You Need](https://mlsanity.com/paper/2106.03253/) (2021)
- [Time Series Classification from Scratch with Deep Neural Networks: A Strong Baseline](https://mlsanity.com/paper/1611.06455/) (2016)
- [A Survey on Heterogeneous Graph Embedding: Methods, Techniques, Applications and Sources](https://mlsanity.com/paper/2011.14867/) (2020)
- [Towards Learning a Universal Non-Semantic Representation of Speech](https://mlsanity.com/paper/2002.12764/) (2020)
- [Evaluating Word Embedding Models: Methods and Experimental Results](https://mlsanity.com/paper/1901.09785/) (2019)
- [Spike2Vec: An Efficient and Scalable Embedding Approach for COVID-19 Spike Sequences](https://mlsanity.com/paper/2109.05019/) (2021)
- [A Comparison of Word Embeddings for the Biomedical Natural Language Processing](https://mlsanity.com/paper/1802.00400/) (2018)
- [A k-mer Based Approach for SARS-CoV-2 Variant Identification](https://mlsanity.com/paper/2108.03465/) (2021)
- [PWM2Vec: An Efficient Embedding Approach for Viral Host Specification from Coronavirus Spike Sequences](https://mlsanity.com/paper/2201.02273/) (2022)
- [Efficient Approximation Algorithms for String Kernel Based Sequence Classification](https://mlsanity.com/paper/1712.04264/) (2017)
- [Deep Learning for Answer Sentence Selection](https://mlsanity.com/paper/1412.1632/) (2014)
- [Robust Representation and Efficient Feature Selection Allows for Effective Clustering of SARS-CoV-2 Variants](https://mlsanity.com/paper/2110.09622/) (2021)
- [Portuguese Word Embeddings: Evaluating on Word Analogies and Natural Language Tasks](https://mlsanity.com/paper/1708.06025/) (2017)
- [MetaGraph2Vec: Complex Semantic Path Augmented Heterogeneous Network Embedding](https://mlsanity.com/paper/1803.02533/) (2018)
- [GaKCo: a Fast GApped k-mer string Kernel using COunting](https://mlsanity.com/paper/1704.07468/) (2017)
- [Informative Initialization and Kernel Selection Improves t-SNE for Biological Sequences](https://mlsanity.com/paper/2211.09263/) (2022)
- [Intelligent Word Embeddings of Free-Text Radiology Reports](https://mlsanity.com/paper/1711.06968/) (2017)

## Abstract (arXiv)

Effective representation of data is crucial in various machine learning tasks, as it captures the underlying structure and context of the data. Embeddings have emerged as a powerful technique for data representation, but evaluating their quality and capacity to preserve structural and contextual information remains a challenge. In this paper, we address this need by proposing a method to measure the textitrepresentation capacity of embeddings. The motivation behind this work stems from the importance of understanding the strengths and limitations of embeddings, enabling researchers and practitioners to make informed decisions in selecting appropriate embedding models for their specific applications. By combining extrinsic evaluation methods, such as classification and clustering, with t-SNE-based neighborhood analysis, such as neighborhood agreement and trustworthiness, we provide a comprehensive assessment of the representation capacity. Additionally, the use of optimization techniques (bayesian optimization) for weight optimization (for classification, clustering, neighborhood agreement, and trustworthiness) ensures an objective and data-driven approach in selecting the optimal combination of metrics. The proposed method not only contributes to advancing the field of embedding evaluation but also empowers researchers and practitioners with a quantitative measure to assess the effectiveness of embeddings in capturing structural and contextual information. For the evaluation, we use 3 real-world biological sequence (proteins and nucleotide) datasets and performed representation capacity analysis of 4 embedding methods from the literature, namely Spike2Vec, Spaced k-mers, PWM2Vec, and AutoEncoder.