# Representation Learning via Variational Bayesian Networks

arXiv: https://arxiv.org/abs/2306.16326 · HTML: https://mlsanity.com/paper/2306.16326/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 62 references that are in the map.
- First posted on arXiv 28 Jun 2023.

## Builds on (most cited first, 29 of 29)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Distributed Representations of Words and Phrases and their Compositionality](https://mlsanity.com/paper/1310.4546/) (2013)
- [Enriching Word Vectors with Subword Information](https://mlsanity.com/paper/1607.04606/) (2016)
- [Neural Graph Collaborative Filtering](https://mlsanity.com/paper/1905.08108/) (2019)
- [Variational Autoencoders for Collaborative Filtering](https://mlsanity.com/paper/1802.05814/) (2018)
- [Knowledge Enhanced Contextual Word Representations](https://mlsanity.com/paper/1909.04164/) (2019)
- [SimLex-999: Evaluating Semantic Models with (Genuine) Similarity Estimation](https://mlsanity.com/paper/1408.3456/) (2014)
- [Poincar'e GloVe: Hyperbolic Word Embeddings](https://mlsanity.com/paper/1810.06546/) (2018)
- [Word Representations via Gaussian Embedding](https://mlsanity.com/paper/1412.6623/) (2014)
- [GlossBERT: BERT for Word Sense Disambiguation with Gloss Knowledge](https://mlsanity.com/paper/1908.07245/) (2019)
- [SenseBERT: Driving Some Sense into BERT](https://mlsanity.com/paper/1908.05646/) (2019)
- [Item2Vec: Neural Item Embedding for Collaborative Filtering](https://mlsanity.com/paper/1603.04259/) (2016)
- [Mimicking Word Embeddings using Subword RNNs](https://mlsanity.com/paper/1707.06961/) (2017)
- [Few-Shot Representation Learning for Out-Of-Vocabulary Words](https://mlsanity.com/paper/1907.00505/) (2019)
- [Generalizing Point Embeddings using the Wasserstein Space of Elliptical Distributions](https://mlsanity.com/paper/1805.07594/) (2018)
- [Attentive Mimicking: Better Word Embeddings by Attending to Informative Contexts](https://mlsanity.com/paper/1904.01617/) (2019)
- [CDLM: Cross-Document Language Modeling](https://mlsanity.com/paper/2101.00406/) (2021)
- [Explainable Recommendations via Attentive Multi-Persona Collaborative Filtering](https://mlsanity.com/paper/2010.07042/) (2020)
- [Scalable Attentive Sentence-Pair Modeling via Distilled Sentence Embedding](https://mlsanity.com/paper/1908.05161/) (2019)
- [Bayesian Neural Word Embedding](https://mlsanity.com/paper/1603.06571/) (2016)
- [Neural Attentive Multiview Machines](https://mlsanity.com/paper/2002.07696/) (2020)
- [Bayesian Hierarchical Words Representation Learning](https://mlsanity.com/paper/2004.07126/) (2020)
- [Embedding Words as Distributions with a Bayesian Skip-gram Model](https://mlsanity.com/paper/1711.11027/) (2017)
- [Attentive Item2Vec: Neural Attentive User Representations](https://mlsanity.com/paper/2002.06205/) (2020)
- [Joint Embedding of Hierarchical Categories and Entities for Concept Categorization and Dataless Classification](https://mlsanity.com/paper/1607.07956/) (2016)
- [Joint Word Representation Learning using a Corpus and a Semantic Lexicon](https://mlsanity.com/paper/1511.06438/) (2015)
- [Sherlock: Sparse Hierarchical Embeddings for Visually-aware One-class Collaborative Filtering](https://mlsanity.com/paper/1604.05813/) (2016)
- [Denoising Word Embeddings by Averaging in a Shared Space](https://mlsanity.com/paper/2106.02954/) (2021)

## Built on by (most cited first, 3 of 3)

- [Deep Integrated Explanations](https://mlsanity.com/paper/2310.15368/) (2023)
- [Learning to Explain: A Model-Agnostic Framework for Explaining Black Box Models](https://mlsanity.com/paper/2310.16584/) (2023)
- [Efficient Discovery and Effective Evaluation of Visual Perceptual Similarity: A Benchmark and Beyond](https://mlsanity.com/paper/2308.14753/) (2023)

## Abstract (arXiv)

We present Variational Bayesian Network (VBN) - a novel Bayesian entity representation learning model that utilizes hierarchical and relational side information and is particularly useful for modeling entities in the ``long-tail'', where the data is scarce. VBN provides better modeling for long-tail entities via two complementary mechanisms: First, VBN employs informative hierarchical priors that enable information propagation between entities sharing common ancestors. Additionally, VBN models explicit relations between entities that enforce complementary structure and consistency, guiding the learned representations towards a more meaningful arrangement in space. Second, VBN represents entities by densities (rather than vectors), hence modeling uncertainty that plays a complementary role in coping with data scarcity. Finally, we propose a scalable Variational Bayes optimization algorithm that enables fast approximate Bayesian inference. We evaluate the effectiveness of VBN on linguistic, recommendations, and medical inference tasks. Our findings show that VBN outperforms other existing methods across multiple datasets, and especially in the long-tail.