# Biomedical Interpretable Entity Representations

arXiv: https://arxiv.org/abs/2106.09502 · HTML: https://mlsanity.com/paper/2106.09502/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 39 references that are in the map.
- First posted on arXiv 17 Jun 2021.

## Builds on (most cited first, 28 of 28)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Billion-scale similarity search with GPUs](https://mlsanity.com/paper/1702.08734/) (2017)
- [Deep contextualized word representations](https://mlsanity.com/paper/1802.05365/) (2018)
- [BioBERT: a pre-trained biomedical language representation model for biomedical text mining](https://mlsanity.com/paper/1901.08746/) (2019)
- [Language Models as Knowledge Bases?](https://mlsanity.com/paper/1909.01066/) (2019)
- [Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation](https://mlsanity.com/paper/1609.08144/) (2016)
- [SciBERT: A Pretrained Language Model for Scientific Text](https://mlsanity.com/paper/1903.10676/) (2019)
- [How Much Knowledge Can You Pack Into the Parameters of a Language Model?](https://mlsanity.com/paper/2002.08910/) (2020)
- [Designing and Interpreting Probes with Control Tasks](https://mlsanity.com/paper/1909.03368/) (2019)
- [ScispaCy: Fast and Robust Models for Biomedical Natural Language Processing](https://mlsanity.com/paper/1902.07669/) (2019)
- [Self-Alignment Pretraining for Biomedical Entity Representations](https://mlsanity.com/paper/2010.11784/) (2020)
- [Information-Theoretic Probing with Minimum Description Length](https://mlsanity.com/paper/2003.12298/) (2020)
- [Deep k-Nearest Neighbors: Towards Confident, Interpretable and Robust Deep Learning](https://mlsanity.com/paper/1803.04765/) (2018)
- [Ultra-Fine Entity Typing](https://mlsanity.com/paper/1807.04905/) (2018)
- [MedMentions: A Large Biomedical Corpus Annotated with UMLS Concepts](https://mlsanity.com/paper/1902.09476/) (2019)
- [Biomedical Entity Representations with Synonym Marginalization](https://mlsanity.com/paper/2005.00239/) (2020)
- [SPINE: SParse Interpretable Neural Embeddings](https://mlsanity.com/paper/1711.08792/) (2017)
- [COMETA: A Corpus for Medical Entity Linking in the Social Media](https://mlsanity.com/paper/2010.03295/) (2020)
- [Sparse Overcomplete Word Vector Representations](https://mlsanity.com/paper/1506.02004/) (2015)
- [DeepType: Multilingual Entity Linking by Neural Type System Evolution](https://mlsanity.com/paper/1802.01021/) (2018)
- [Learning to Denoise Distantly-Labeled Data for Entity Typing](https://mlsanity.com/paper/1905.01566/) (2019)
- [Interpretable Entity Representations through Large-Scale Typing](https://mlsanity.com/paper/2005.00147/) (2020)
- [Hierarchical Losses and New Resources for Fine-grained Entity Typing and Linking](https://mlsanity.com/paper/1807.05127/) (2018)
- [SLING: A framework for frame semantic parsing](https://mlsanity.com/paper/1710.07032/) (2017)
- [Medical Entity Linking using Triplet Network](https://mlsanity.com/paper/2012.11164/) (2020)
- [EntEval: A Holistic Evaluation Benchmark for Entity Representations](https://mlsanity.com/paper/1909.00137/) (2019)
- [Transformation of Dense and Sparse Text Representations](https://mlsanity.com/paper/1911.02914/) (2019)

## Built on by (most cited first, 4 of 4)

- [From Explainable to Interpretable Deep Learning for Natural Language Processing in Healthcare: How Far from Reality?](https://mlsanity.com/paper/2403.11894/) (2024)
- [Towards Explainability in Legal Outcome Prediction Models](https://mlsanity.com/paper/2403.16852/) (2024)
- [Intermediate Entity-based Sparse Interpretable Representation Learning](https://mlsanity.com/paper/2212.01641/) (2022)
- [Asking the Right Questions: Ontology-Grounded Interpretable Embeddings for Biomedical Text](https://mlsanity.com/paper/2603.01690/) (2026)

## Abstract (arXiv)

Pre-trained language models induce dense entity representations that offer strong performance on entity-centric NLP tasks, but such representations are not immediately interpretable. This can be a barrier to model uptake in important domains such as biomedicine. There has been recent work on general interpretable representation learning (Onoe and Durrett, 2020), but these domain-agnostic representations do not readily transfer to the important domain of biomedicine. In this paper, we create a new entity type system and training set from a large corpus of biomedical texts by mapping entities to concepts in a medical ontology, and from these to Wikipedia pages whose categories are our types. From this mapping we derive Biomedical Interpretable Entity Representations(BIERs), in which dimensions correspond to fine-grained entity types, and values are predicted probabilities that a given entity is of the corresponding type. We propose a novel method that exploits BIER's final sparse and intermediate dense representations to facilitate model and entity type debugging. We show that BIERs achieve strong performance in biomedical tasks including named entity disambiguation and entity label classification, and we provide error analysis to highlight the utility of their interpretability, particularly in low-supervision settings. Finally, we provide our induced 68K biomedical type system, the corresponding 37 million triples of derived data used to train BIER models and our best performing model.