# Modelling Commonsense Commonalities with Multi-Facet Concept Embeddings

arXiv: https://arxiv.org/abs/2403.16984 · HTML: https://mlsanity.com/paper/2403.16984/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 26 of its 50 references that are in the map.
- First posted on arXiv 25 Mar 2024.

## Builds on (most cited first, 26 of 26)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Challenging Common Assumptions in the Unsupervised Learning of Disentangled Representations](https://mlsanity.com/paper/1811.12359/) (2018)
- [InfoGAN: Interpretable Representation Learning by Information Maximizing Generative Adversarial Nets](https://mlsanity.com/paper/1606.03657/) (2016)
- [How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddings](https://mlsanity.com/paper/1909.00512/) (2019)
- [Disentangling by Factorising](https://mlsanity.com/paper/1802.05983/) (2018)
- [Probing Pretrained Language Models for Lexical Semantics](https://mlsanity.com/paper/2010.05731/) (2020)
- [Adaptive Cross-Modal Few-Shot Learning](https://mlsanity.com/paper/1902.07104/) (2019)
- [Fast, Effective, and Self-Supervised: Transforming Masked Language Models into Universal Lexical and Sentence Encoders](https://mlsanity.com/paper/2104.08027/) (2021)
- [Ultra-Fine Entity Typing](https://mlsanity.com/paper/1807.04905/) (2018)
- [Do Neural Language Representations Learn Physical Commonsense?](https://mlsanity.com/paper/1908.02899/) (2019)
- [GenericsKB: A Knowledge Base of Generic Statements](https://mlsanity.com/paper/2005.00660/) (2020)
- [An Empirical Study on Information Extraction using Large Language Models](https://mlsanity.com/paper/2305.14450/) (2023)
- [CAR: Conceptualization-Augmented Reasoner for Zero-Shot Commonsense Question Answering](https://mlsanity.com/paper/2305.14869/) (2023)
- [CAT: A Contextualized Conceptualization and Instantiation Framework for Commonsense Reasoning](https://mlsanity.com/paper/2305.04808/) (2023)
- [Imposing Label-Relational Inductive Bias for Extremely Fine-Grained Entity Typing](https://mlsanity.com/paper/1903.02591/) (2019)
- [MirrorWiC: On Eliciting Word-in-Context Representations from Pretrained Language Models](https://mlsanity.com/paper/2109.09237/) (2021)
- [Unified Semantic Typing with Meaningful Label Inference](https://mlsanity.com/paper/2205.01826/) (2022)
- [Acquiring and Modelling Abstract Commonsense Knowledge via Conceptualization](https://mlsanity.com/paper/2206.01532/) (2022)
- [Automatic Noisy Label Correction for Fine-Grained Entity Typing](https://mlsanity.com/paper/2205.03011/) (2022)
- [Learning Disentangled Representations of Texts with Application to Biomedical Abstracts](https://mlsanity.com/paper/1804.07212/) (2018)
- [Modelling Commonsense Properties using Pre-Trained Bi-Encoders](https://mlsanity.com/paper/2210.02771/) (2022)
- [Distilling Semantic Concept Embeddings from Contrastively Fine-Tuned Language Models](https://mlsanity.com/paper/2305.09785/) (2023)
- [Ultra-Fine Entity Typing with Prior Knowledge about Labels: A Simple Clustering Based Strategy](https://mlsanity.com/paper/2305.12802/) (2023)
- [Automated Generation of Multilingual Clusters for the Evaluation of Distributed Representations](https://mlsanity.com/paper/1611.01547/) (2016)
- [Cabbage Sweeter than Cake? Analysing the Potential of Large Language Models for Learning Conceptual Spaces](https://mlsanity.com/paper/2310.05481/) (2023)
- [What do Deck Chairs and Sun Hats Have in Common? Uncovering Shared Properties in Large Concept Vocabularies](https://mlsanity.com/paper/2310.14793/) (2023)

## Abstract (arXiv)

Concept embeddings offer a practical and efficient mechanism for injecting commonsense knowledge into downstream tasks. Their core purpose is often not to predict the commonsense properties of concepts themselves, but rather to identify commonalities, i.e. sets of concepts which share some property of interest. Such commonalities are the basis for inductive generalisation, hence high-quality concept embeddings can make learning easier and more robust. Unfortunately, standard embeddings primarily reflect basic taxonomic categories, making them unsuitable for finding commonalities that refer to more specific aspects (e.g. the colour of objects or the materials they are made of). In this paper, we address this limitation by explicitly modelling the different facets of interest when learning concept embeddings. We show that this leads to embeddings which capture a more diverse range of commonsense properties, and consistently improves results in downstream tasks such as ultra-fine entity typing and ontology completion.