# SenteCon: Leveraging Lexicons to Learn Human-Interpretable Language Representations

arXiv: https://arxiv.org/abs/2305.14728 · HTML: https://mlsanity.com/paper/2305.14728/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 50 references that are in the map.
- First posted on arXiv 24 May 2023.

## Builds on (most cited first, 26 of 26)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization](https://mlsanity.com/paper/1610.02391/) (2016)
- [Efficient Estimation of Word Representations in Vector Space](https://mlsanity.com/paper/1301.3781/) (2013)
- [DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter](https://mlsanity.com/paper/1910.01108/) (2019)
- [Explainable Artificial Intelligence (XAI): Concepts, Taxonomies, Opportunities and Challenges toward Responsible AI](https://mlsanity.com/paper/1910.10045/) (2019)
- [MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformers](https://mlsanity.com/paper/2002.10957/) (2020)
- [Towards A Rigorous Science of Interpretable Machine Learning](https://mlsanity.com/paper/1702.08608/) (2017)
- [Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddings](https://mlsanity.com/paper/1607.06520/) (2016)
- [What Does BERT Look At? An Analysis of BERT's Attention](https://mlsanity.com/paper/1906.04341/) (2019)
- [MPNet: Masked and Permuted Pre-training for Language Understanding](https://mlsanity.com/paper/2004.09297/) (2020)
- [MELD: A Multimodal Multi-Party Dataset for Emotion Recognition in Conversations](https://mlsanity.com/paper/1810.02508/) (2018)
- [Visualizing and Measuring the Geometry of BERT](https://mlsanity.com/paper/1906.02715/) (2019)
- [Causal Inference in Natural Language Processing: Estimation, Prediction, Interpretation and Beyond](https://mlsanity.com/paper/2109.00725/) (2021)
- [PubMed 200k RCT: a Dataset for Sequential Sentence Classification in Medical Abstracts](https://mlsanity.com/paper/1710.06071/) (2017)
- [Fairness in Deep Learning: A Computational Perspective](https://mlsanity.com/paper/1908.08843/) (2019)
- [An Interpretability Illusion for BERT](https://mlsanity.com/paper/2104.07143/) (2021)
- [Does BERT Make Any Sense? Interpretable Word Sense Disambiguation with Contextualized Embeddings](https://mlsanity.com/paper/1909.10430/) (2019)
- [Empath: Understanding Topic Signals in Large-Scale Text](https://mlsanity.com/paper/1602.06979/) (2016)
- [What are the biases in my word embedding?](https://mlsanity.com/paper/1812.08769/) (2018)
- [Interpretable Entity Representations through Large-Scale Typing](https://mlsanity.com/paper/2005.00147/) (2020)
- [Contextual-Lexicon Approach for Abusive Language Detection](https://mlsanity.com/paper/2104.12265/) (2021)
- [Race, Ethnicity and National Origin-based Discrimination in Social Media and Hate Crimes Across 100 U.S. Cities](https://mlsanity.com/paper/1902.00119/) (2019)
- [BERT as a Teacher: Contextual Embeddings for Sequence-Level Reward](https://mlsanity.com/paper/2003.02738/) (2020)
- [Towards Grad-CAM Based Explainability in a Legal Text Processing Pipeline](https://mlsanity.com/paper/2012.09603/) (2020)
- [Toward Multimodal Modeling of Emotional Expressiveness](https://mlsanity.com/paper/2009.00001/) (2020)

## Built on by (most cited first, 2 of 2)

- [Isolated Causal Effects of Natural Language](https://mlsanity.com/paper/2410.14812/) (2024)
- [Omitted Variable Bias in Language Models Under Distribution Shift](https://mlsanity.com/paper/2602.16784/) (2026)

## Abstract (arXiv)

Although deep language representations have become the dominant form of language featurization in recent years, in many settings it is important to understand a model's decision-making process. This necessitates not only an interpretable model but also interpretable features. In particular, language must be featurized in a way that is interpretable while still characterizing the original text well. We present SenteCon, a method for introducing human interpretability in deep language representations. Given a passage of text, SenteCon encodes the text as a layer of interpretable categories in which each dimension corresponds to the relevance of a specific category. Our empirical evaluations indicate that encoding language with SenteCon provides high-level interpretability at little to no cost to predictive performance on downstream tasks. Moreover, we find that SenteCon outperforms existing interpretable language representations with respect to both its downstream performance and its agreement with human characterizations of the text.