# Enhancing Interpretable Clauses Semantically using Pretrained Word Representation

arXiv: https://arxiv.org/abs/2104.06901 · HTML: https://mlsanity.com/paper/2104.06901/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 38 references that are in the map.
- First posted on arXiv 14 Apr 2021.

## Builds on (most cited first, 18 of 18)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Distributed Representations of Words and Phrases and their Compositionality](https://mlsanity.com/paper/1310.4546/) (2013)
- [XLNet: Generalized Autoregressive Pretraining for Language Understanding](https://mlsanity.com/paper/1906.08237/) (2019)
- [Convolutional Neural Networks on Graphs with Fast Localized Spectral Filtering](https://mlsanity.com/paper/1606.09375/) (2016)
- [Enriching Word Vectors with Subword Information](https://mlsanity.com/paper/1607.04606/) (2016)
- [Convolutional Neural Networks for Sentence Classification](https://mlsanity.com/paper/1408.5882/) (2014)
- [Bag of Tricks for Efficient Text Classification](https://mlsanity.com/paper/1607.01759/) (2016)
- [Distributed Representations of Sentences and Documents](https://mlsanity.com/paper/1405.4053/) (2014)
- [BAE: BERT-based Adversarial Examples for Text Classification](https://mlsanity.com/paper/2004.01970/) (2020)
- [Recurrent Neural Network for Text Classification with Multi-Task Learning](https://mlsanity.com/paper/1605.05101/) (2016)
- [PTE: Predictive Text Embedding through Large-scale Heterogeneous Text Networks](https://mlsanity.com/paper/1508.00200/) (2015)
- [The Tsetlin Machine -- A Game Theoretic Bandit Driven Approach to Optimal Pattern Recognition with Propositional Logic](https://mlsanity.com/paper/1804.01508/) (2018)
- [Baseline Needs More Love: On Simple Word-Embedding-Based Models and Associated Pooling Mechanisms](https://mlsanity.com/paper/1805.09843/) (2018)
- [The Convolutional Tsetlin Machine](https://mlsanity.com/paper/1905.09688/) (2019)
- [Using the Tsetlin Machine to Learn Human-Interpretable Rules for High-Accuracy Text Categorization with Medical Applications](https://mlsanity.com/paper/1809.04547/) (2018)
- [Measuring the Novelty of Natural Language Text Using the Conjunctive Clauses of a Tsetlin Machine Text Classifier](https://mlsanity.com/paper/2011.08755/) (2020)
- [A Scheme for Continuous Input to the Tsetlin Machine with Applications to Forecasting Disease Outbreaks](https://mlsanity.com/paper/1905.04199/) (2019)
- [Label-guided Learning for Text Classification](https://mlsanity.com/paper/2002.10772/) (2020)

## Built on by (most cited first, 10 of 10)

- [Building Concise Logical Patterns by Constraining Tsetlin Machine Clause Size](https://mlsanity.com/paper/2301.08190/) (2023)
- [What happens before and after: Multi-Event Commonsense in Event Coreference Resolution](https://mlsanity.com/paper/2302.09715/) (2023)
- [Word-level Human Interpretable Scoring Mechanism for Novel Text Detection Using Tsetlin Machines](https://mlsanity.com/paper/2105.04708/) (2021)
- [On the Convergence of Tsetlin Machines for the AND and the OR Operators](https://mlsanity.com/paper/2109.09488/) (2021)
- [Interpretable Tsetlin Machine-based Premature Ventricular Contraction Identification](https://mlsanity.com/paper/2301.10181/) (2023)
- [The Tsetlin Machine Goes Deep: Logical Learning and Reasoning With Graphs](https://mlsanity.com/paper/2507.14874/) (2025)
- [Omni TM-AE: A Scalable and Interpretable Embedding Model Using the Full Tsetlin Machine State Space](https://mlsanity.com/paper/2505.16386/) (2025)
- [Pruning Literals for Highly Efficient Explainability at Word Level](https://mlsanity.com/paper/2411.04557/) (2024)
- [LLM-Guided Semantic Bootstrapping for Interpretable Text Classification with Tsetlin Machines](https://mlsanity.com/paper/2604.12223/) (2026)
- [Clusters are All You Need: Pre-Training the Tsetlin Machine with Semantic Clusters from Language Models for Interpretability](https://mlsanity.com/paper/2606.19815/) (2026)

## Abstract (arXiv)

Tsetlin Machine (TM) is an interpretable pattern recognition algorithm based on propositional logic, which has demonstrated competitive performance in many Natural Language Processing (NLP) tasks, including sentiment analysis, text classification, and Word Sense Disambiguation. To obtain human-level interpretability, legacy TM employs Boolean input features such as bag-of-words (BOW). However, the BOW representation makes it difficult to use any pre-trained information, for instance, word2vec and GloVe word representations. This restriction has constrained the performance of TM compared to deep neural networks (DNNs) in NLP. To reduce the performance gap, in this paper, we propose a novel way of using pre-trained word representations for TM. The approach significantly enhances the performance and interpretability of TM. We achieve this by extracting semantically related words from pre-trained word representations as input features to the TM. Our experiments show that the accuracy of the proposed approach is significantly higher than the previous BOW-based TM, reaching the level of DNN-based models.