# Expanding Knowledge Graphs with Humans in the Loop

arXiv: https://arxiv.org/abs/2212.05189 · HTML: https://mlsanity.com/paper/2212.05189/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 36 of its 82 references that are in the map.
- First posted on arXiv 10 Dec 2022.

## Builds on (most cited first, 30 of 36)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Distributed Representations of Words and Phrases and their Compositionality](https://mlsanity.com/paper/1310.4546/) (2013)
- [FaceNet: A Unified Embedding for Face Recognition and Clustering](https://mlsanity.com/paper/1503.03832/) (2015)
- [MS MARCO: A Human Generated MAchine Reading COmprehension Dataset](https://mlsanity.com/paper/1611.09268/) (2016)
- [Enriching Word Vectors with Subword Information](https://mlsanity.com/paper/1607.04606/) (2016)
- [MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformers](https://mlsanity.com/paper/2002.10957/) (2020)
- [Pretrained Transformers for Text Ranking: BERT and Beyond](https://mlsanity.com/paper/2010.06467/) (2020)
- [Advances in Pre-Training Distributed Word Representations](https://mlsanity.com/paper/1712.09405/) (2017)
- [MMKG: Multi-Modal Knowledge Graphs](https://mlsanity.com/paper/1903.05485/) (2019)
- [The Algorithmic Automation Problem: Prediction, Triage, and Human Effort](https://mlsanity.com/paper/1903.12220/) (2019)
- [Baseline Needs More Love: On Simple Word-Embedding-Based Models and Associated Pooling Mechanisms](https://mlsanity.com/paper/1805.09843/) (2018)
- [TaxoGen: Unsupervised Topic Taxonomy Construction by Adaptive Term Embedding and Clustering](https://mlsanity.com/paper/1812.09551/) (2018)
- [Human-Algorithm Collaboration: Achieving Complementarity and Avoiding Unfairness](https://mlsanity.com/paper/2202.08821/) (2022)
- [TaxoExpan: Self-supervised Taxonomy Expansion with Position-Enhanced Graph Neural Network](https://mlsanity.com/paper/2001.09522/) (2020)
- [Human-AI Collaboration with Bandit Feedback](https://mlsanity.com/paper/2105.10614/) (2021)
- [HiExpan: Task-Guided Taxonomy Construction by Hierarchical Tree Expansion](https://mlsanity.com/paper/1910.08194/) (2019)
- [STEAM: Self-Supervised Taxonomy Expansion with Mini-Paths](https://mlsanity.com/paper/2006.10217/) (2020)
- [AutoKnow: Self-Driving Knowledge Collection for Products of Thousands of Types](https://mlsanity.com/paper/2006.13473/) (2020)
- [Octet: Online Catalog Taxonomy Enrichment with Self-Supervision](https://mlsanity.com/paper/2006.10276/) (2020)
- [Taxonomy Completion via Triplet Matching Network](https://mlsanity.com/paper/2101.01896/) (2021)
- [Enhancing Taxonomy Completion with Concept Generation via Fusing Relational Representations](https://mlsanity.com/paper/2106.02974/) (2021)
- [Algorithmic Fairness in Business Analytics: Directions for Research and Practice](https://mlsanity.com/paper/2207.10991/) (2022)
- [End-to-End Reinforcement Learning for Automatic Taxonomy Induction](https://mlsanity.com/paper/1805.04044/) (2018)
- [Triplet Distillation for Deep Face Recognition](https://mlsanity.com/paper/1905.04457/) (2019)
- [The Best Decisions Are Not the Best Advice: Making Adherence-Aware Recommendations](https://mlsanity.com/paper/2209.01874/) (2022)
- [Enquire One's Parent and Child Before Decision: Fully Exploit Hierarchical Structure for Self-Supervised Taxonomy Expansion](https://mlsanity.com/paper/2101.11268/) (2021)
- [When Does Uncertainty Matter?: Understanding the Impact of Predictive Uncertainty in ML Assisted Decision Making](https://mlsanity.com/paper/2011.06167/) (2020)
- [Improving Human Sequential Decision-Making with Reinforcement Learning](https://mlsanity.com/paper/2108.08454/) (2021)
- [Algorithmic Transparency with Strategic Users](https://mlsanity.com/paper/2008.09283/) (2020)

## Built on by (most cited first, 3 of 3)

- [NASA Science Mission Directorate Knowledge Graph Discovery](https://mlsanity.com/paper/2303.10871/) (2023)
- [ASGM-KG: Unveiling Alluvial Gold Mining Through Knowledge Graphs](https://mlsanity.com/paper/2408.08972/) (2024)
- [Towards a Reliable Offline Personal AI Assistant for Long Duration Spaceflight](https://mlsanity.com/paper/2410.16397/) (2024)

## Abstract (arXiv)

Curated knowledge graphs encode domain expertise and improve the performance of recommendation, segmentation, ad targeting, and other machine learning systems in several domains. As new concepts emerge in a domain, knowledge graphs must be expanded to preserve machine learning performance. Manually expanding knowledge graphs, however, is infeasible at scale. In this work, we propose a method for knowledge graph expansion with humans-in-the-loop. Concretely, given a knowledge graph, our method predicts the "parents" of new concepts to be added to this graph for further verification by human experts. We show that our method is both accurate and provably "human-friendly". Specifically, we prove that our method predicts parents that are "near" concepts' true parents in the knowledge graph, even when the predictions are incorrect. We then show, with a controlled experiment, that satisfying this property increases both the speed and the accuracy of the human-algorithm collaboration. We further evaluate our method on a knowledge graph from Pinterest and show that it outperforms competing methods on both accuracy and human-friendliness. Upon deployment in production at Pinterest, our method reduced the time needed for knowledge graph expansion by ~400% (compared to manual expansion), and contributed to a subsequent increase in ad revenue of 20%.