# Word2Box: Capturing Set-Theoretic Semantics of Words using Box Embeddings

arXiv: https://arxiv.org/abs/2106.14361 · HTML: https://mlsanity.com/paper/2106.14361/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 54 references that are in the map.
- First posted on arXiv 28 Jun 2021.

## Builds on (most cited first, 23 of 23)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Distributed Representations of Words and Phrases and their Compositionality](https://mlsanity.com/paper/1310.4546/) (2013)
- [Deep contextualized word representations](https://mlsanity.com/paper/1802.05365/) (2018)
- [Enriching Word Vectors with Subword Information](https://mlsanity.com/paper/1607.04606/) (2016)
- [Poincar'e Embeddings for Learning Hierarchical Representations](https://mlsanity.com/paper/1705.08039/) (2017)
- [SimLex-999: Evaluating Semantic Models with (Genuine) Similarity Estimation](https://mlsanity.com/paper/1408.3456/) (2014)
- [Query2box: Reasoning over Knowledge Graphs in Vector Space using Box Embeddings](https://mlsanity.com/paper/2002.05969/) (2020)
- [Hyperbolic Entailment Cones for Learning Hierarchical Embeddings](https://mlsanity.com/paper/1804.01882/) (2018)
- [Poincar'e GloVe: Hyperbolic Word Embeddings](https://mlsanity.com/paper/1810.06546/) (2018)
- [Beta Embeddings for Multi-Hop Logical Reasoning in Knowledge Graphs](https://mlsanity.com/paper/2010.11465/) (2020)
- [Order-Embeddings of Images and Language](https://mlsanity.com/paper/1511.06361/) (2015)
- [BoxE: A Box Embedding Model for Knowledge Base Completion](https://mlsanity.com/paper/2007.06267/) (2020)
- [Word Representations via Gaussian Embedding](https://mlsanity.com/paper/1412.6623/) (2014)
- [Probabilistic Embedding of Knowledge Graphs with Box Lattice Measures](https://mlsanity.com/paper/1805.06627/) (2018)
- [Neural Embeddings of Graphs in Hyperbolic Space](https://mlsanity.com/paper/1705.10359/) (2017)
- [Modeling Fine-Grained Entity Types with Box Embeddings](https://mlsanity.com/paper/2101.00345/) (2021)
- [Improving Local Identifiability in Probabilistic Box Embeddings](https://mlsanity.com/paper/2010.04831/) (2020)
- [Probabilistic Box Embeddings for Uncertain Knowledge Graph Reasoning](https://mlsanity.com/paper/2104.04597/) (2021)
- [Hyperbolic Disk Embeddings for Directed Acyclic Graphs](https://mlsanity.com/paper/1902.04335/) (2019)
- [Hierarchical Density Order Embeddings](https://mlsanity.com/paper/1804.09843/) (2018)
- [Modelling Lexical Ambiguity with Density Matrices](https://mlsanity.com/paper/2010.05670/) (2020)
- [Don't Settle for Average, Go for the Max: Fuzzy Sets and Max-Pooled Word Vectors](https://mlsanity.com/paper/1904.13264/) (2019)

## Built on by (most cited first, 11 of 11)

- [Word Sense Disambiguation as a Game of Neurosymbolic Darts](https://mlsanity.com/paper/2307.16663/) (2023)
- [Task2Box: Box Embeddings for Modeling Asymmetric Task Relationships](https://mlsanity.com/paper/2403.17173/) (2024)
- [SetCSE: Set Operations using Contrastive Learning of Sentence Embeddings](https://mlsanity.com/paper/2404.17606/) (2024)
- [FUSE: Measure-Theoretic Compact Fuzzy Set Representation for Taxonomy Expansion](https://mlsanity.com/paper/2506.08409/) (2025)
- [Insert or Attach: Taxonomy Completion via Box Embedding](https://mlsanity.com/paper/2305.11004/) (2023)
- [Sentence Representations via Gaussian Embedding](https://mlsanity.com/paper/2305.12990/) (2023)
- [InBox: Recommendation with Knowledge Graph using Interest Box Embedding](https://mlsanity.com/paper/2403.12649/) (2024)
- [When Box Meets Graph Neural Network in Tag-aware Recommendation](https://mlsanity.com/paper/2406.12020/) (2024)
- [LARE: Latent Augmentation using Regional Embedding with Vision-Language Model](https://mlsanity.com/paper/2409.12597/) (2024)
- [Answering Compositional Queries with Set-Theoretic Embeddings](https://mlsanity.com/paper/2306.04133/) (2023)
- [What Machine Learning Tells Us About the Mathematical Structure of Concepts](https://mlsanity.com/paper/2408.15507/) (2024)

## Abstract (arXiv)

Learning representations of words in a continuous space is perhaps the most fundamental task in NLP, however words interact in ways much richer than vector dot product similarity can provide. Many relationships between words can be expressed set-theoretically, for example, adjective-noun compounds (eg. "red cars"subseteq"cars") and homographs (eg. "tongue"cap"body" should be similar to "mouth", while "tongue"cap"language" should be similar to "dialect") have natural set-theoretic interpretations. Box embeddings are a novel region-based representation which provide the capability to perform these set-theoretic operations. In this work, we provide a fuzzy-set interpretation of box embeddings, and learn box representations of words using a set-theoretic training objective. We demonstrate improved performance on various word similarity tasks, particularly on less common words, and perform a quantitative and qualitative analysis exploring the additional unique expressivity provided by Word2Box.