# DOCENT: Learning Self-Supervised Entity Representations from Large Document Collections

arXiv: https://arxiv.org/abs/2102.13247 · HTML: https://mlsanity.com/paper/2102.13247/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 46 references that are in the map.
- First posted on arXiv 26 Feb 2021.

## Builds on (most cited first, 19 of 19)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Distributed Representations of Words and Phrases and their Compositionality](https://mlsanity.com/paper/1310.4546/) (2013)
- [Deep contextualized word representations](https://mlsanity.com/paper/1802.05365/) (2018)
- [BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions](https://mlsanity.com/paper/1905.10044/) (2019)
- [Ups and Downs: Modeling the Visual Evolution of Fashion Trends with One-Class Collaborative Filtering](https://mlsanity.com/paper/1602.01585/) (2016)
- [ERNIE: Enhanced Representation through Knowledge Integration](https://mlsanity.com/paper/1904.09223/) (2019)
- [Knowledge Enhanced Contextual Word Representations](https://mlsanity.com/paper/1909.04164/) (2019)
- [Pre-training Tasks for Embedding-based Large-scale Retrieval](https://mlsanity.com/paper/2002.03932/) (2020)
- [Zero-Shot Entity Linking by Reading Entity Descriptions](https://mlsanity.com/paper/1906.07348/) (2019)
- [Learning Dense Representations for Entity Retrieval](https://mlsanity.com/paper/1909.10506/) (2019)
- [Learned in Translation: Contextualized Word Vectors](https://mlsanity.com/paper/1708.00107/) (2017)
- [Entities as Experts: Sparse Memory Access with Entity Supervision](https://mlsanity.com/paper/2004.07202/) (2020)
- [Scalable Zero-shot Entity Linking with Dense Entity Retrieval](https://mlsanity.com/paper/1911.03814/) (2019)
- [Joint Learning of the Embedding of Words and Entities for Named Entity Disambiguation](https://mlsanity.com/paper/1601.01343/) (2016)
- [Learning Cross-Context Entity Representations from Text](https://mlsanity.com/paper/2001.03765/) (2020)
- [Learning Distributed Representations of Texts and Entities from Knowledge Base](https://mlsanity.com/paper/1705.02494/) (2017)
- [Beyond Word Embeddings: Learning Entity and Concept Representations from Large Scale Knowledge Bases](https://mlsanity.com/paper/1801.00388/) (2018)
- [EntEval: A Holistic Evaluation Benchmark for Entity Representations](https://mlsanity.com/paper/1909.00137/) (2019)

## Built on by (most cited first, 13 of 13)

- [Large Language Models are Competitive Near Cold-start Recommenders for Language- and Item-based Preferences](https://mlsanity.com/paper/2307.14225/) (2023)
- [Scientific Language Models for Biomedical Knowledge Base Completion: An Empirical Study](https://mlsanity.com/paper/2106.09700/) (2021)
- [Automate Knowledge Concept Tagging on Math Questions with LLMs](https://mlsanity.com/paper/2403.17281/) (2024)
- [Knowledge Tagging System on Math Questions via LLMs with Flexible Demonstration Retriever](https://mlsanity.com/paper/2406.13885/) (2024)
- [Knowledge Tagging with Large Language Model based Multi-Agent System](https://mlsanity.com/paper/2409.08406/) (2024)
- [SINGA-Easy: An Easy-to-Use Framework for MultiModal Analysis](https://mlsanity.com/paper/2108.02572/) (2021)
- [VisualTextRank: Unsupervised Graph-based Content Extraction for Automating Ad Text to Image Search](https://mlsanity.com/paper/2108.02725/) (2021)
- [Text is NOT Enough: Integrating Visual Impressions into Open-domain Dialogue Generation](https://mlsanity.com/paper/2109.05778/) (2021)
- [Discovering Personalized Semantics for Soft Attributes in Recommender Systems using Concept Activation Vectors](https://mlsanity.com/paper/2202.02830/) (2022)
- [Reinforced MOOCs Concept Recommendation in Heterogeneous Information Networks](https://mlsanity.com/paper/2203.11011/) (2022)
- [Explanation-Guided Fairness Testing through Genetic Algorithm](https://mlsanity.com/paper/2205.08335/) (2022)
- [Using Graph Neural Networks for Program Termination](https://mlsanity.com/paper/2207.14648/) (2022)
- [Tribe or Not? Critical Inspection of Group Differences Using TribalGram](https://mlsanity.com/paper/2303.09664/) (2023)

## Abstract (arXiv)

This paper explores learning rich self-supervised entity representations from large amounts of the associated text. Once pre-trained, these models become applicable to multiple entity-centric tasks such as ranked retrieval, knowledge base completion, question answering, and more. Unlike other methods that harvest self-supervision signals based merely on a local context within a sentence, we radically expand the notion of context to include any available text related to an entity. This enables a new class of powerful, high-capacity representations that can ultimately distill much of the useful information about an entity from multiple text sources, without any human supervision. We present several training strategies that, unlike prior approaches, learn to jointly predict words and entities -- strategies we compare experimentally on downstream tasks in the TV-Movies domain, such as MovieLens tag prediction from user reviews and natural language movie search. As evidenced by results, our models match or outperform competitive baselines, sometimes with little or no fine-tuning, and can scale to very large corpora. Finally, we make our datasets and pre-trained models publicly available. This includes Reviews2Movielens (see https://goo.gle/research-docent ), mapping the up to 1B word corpus of Amazon movie reviews (He and McAuley, 2016) to MovieLens tags (Harper and Konstan, 2016), as well as Reddit Movie Suggestions (see https://urikz.github.io/docent ) with natural language queries and corresponding community recommendations.