# Drop Clause: Enhancing Performance, Interpretability and Robustness of the Tsetlin Machine

arXiv: https://arxiv.org/abs/2105.14506 · HTML: https://mlsanity.com/paper/2105.14506/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 43 references that are in the map.
- First posted on arXiv 30 May 2021.

## Builds on (most cited first, 24 of 24)

- [Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization](https://mlsanity.com/paper/1610.02391/) (2016)
- [Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps](https://mlsanity.com/paper/1312.6034/) (2013)
- [Deep contextualized word representations](https://mlsanity.com/paper/1802.05365/) (2018)
- [Convolutional Neural Networks for Sentence Classification](https://mlsanity.com/paper/1408.5882/) (2014)
- [Attention is not Explanation](https://mlsanity.com/paper/1902.10186/) (2019)
- [Network Dissection: Quantifying Interpretability of Deep Visual Representations](https://mlsanity.com/paper/1704.05796/) (2017)
- [Interpretation of Neural Networks is Fragile](https://mlsanity.com/paper/1710.10547/) (2017)
- [Is Attention Interpretable?](https://mlsanity.com/paper/1906.03731/) (2019)
- [An Intriguing Failing of Convolutional Neural Networks and the CoordConv Solution](https://mlsanity.com/paper/1807.03247/) (2018)
- [InterpretML: A Unified Framework for Machine Learning Interpretability](https://mlsanity.com/paper/1909.09223/) (2019)
- [Maxout Networks](https://mlsanity.com/paper/1302.4389/) (2013)
- [Interpretable Convolutional Neural Networks](https://mlsanity.com/paper/1710.00935/) (2017)
- [MNIST-C: A Robustness Benchmark for Computer Vision](https://mlsanity.com/paper/1906.02337/) (2019)
- [PTE: Predictive Text Embedding through Large-scale Heterogeneous Text Networks](https://mlsanity.com/paper/1508.00200/) (2015)
- [Verifying Properties of Binarized Deep Neural Networks](https://mlsanity.com/paper/1709.06662/) (2017)
- [The Tsetlin Machine -- A Game Theoretic Bandit Driven Approach to Optimal Pattern Recognition with Propositional Logic](https://mlsanity.com/paper/1804.01508/) (2018)
- [The Convolutional Tsetlin Machine](https://mlsanity.com/paper/1905.09688/) (2019)
- [Efficient Exact Verification of Binarized Neural Networks](https://mlsanity.com/paper/2005.03597/) (2020)
- [Using the Tsetlin Machine to Learn Human-Interpretable Rules for High-Accuracy Text Categorization with Medical Applications](https://mlsanity.com/paper/1809.04547/) (2018)
- [On the Convergence of Tsetlin Machines for the IDENTITY- and NOT Operators](https://mlsanity.com/paper/2007.14268/) (2020)
- [Extending the Tsetlin Machine With Integer-Weighted Clauses for Increased Interpretability](https://mlsanity.com/paper/2005.05131/) (2020)
- [On the Convergence of Tsetlin Machines for the XOR Operator](https://mlsanity.com/paper/2101.02547/) (2021)
- [Low-Power Audio Keyword Spotting using Tsetlin Machines](https://mlsanity.com/paper/2101.11336/) (2021)
- [Patchnet: Interpretable Neural Networks for Image Classification](https://mlsanity.com/paper/1705.08078/) (2017)

## Built on by (most cited first, 7 of 7)

- [Tsetlin Machine for Solving Contextual Bandit Problems](https://mlsanity.com/paper/2202.01914/) (2022)
- [Exploring Effects of Hyperdimensional Vectors for Tsetlin Machines](https://mlsanity.com/paper/2406.02648/) (2024)
- [MATADOR: Automated System-on-Chip Tsetlin Machine Design Generation for Edge Applications](https://mlsanity.com/paper/2403.10538/) (2024)
- [On the Equivalence of the Weighted Tsetlin Machine and the Perceptron](https://mlsanity.com/paper/2212.13634/) (2022)
- [The Sparse Tsetlin Machine: Sparse Representation with Active Literals](https://mlsanity.com/paper/2405.02375/) (2024)
- [Verifying Properties of Tsetlin Machines](https://mlsanity.com/paper/2303.14464/) (2023)
- [Generating Bayesian Network Models from Data Using Tsetlin Machines](https://mlsanity.com/paper/2305.10538/) (2023)

## Abstract (arXiv)

In this article, we introduce a novel variant of the Tsetlin machine (TM) that randomly drops clauses, the key learning elements of a TM. In effect, TM with drop clause ignores a random selection of the clauses in each epoch, selected according to a predefined probability. In this way, additional stochasticity is introduced in the learning phase of TM. To explore the effects drop clause has on accuracy, training time, interpretability and robustness, we conduct extensive experiments on nine benchmark datasets in natural language processing~(NLP) (IMDb, R8, R52, MR and TREC) and image classification (MNIST, Fashion MNIST, CIFAR-10 and CIFAR-100). Our proposed model outperforms baseline machine learning algorithms by a wide margin and achieves competitive performance in comparison with recent deep learning model such as BERT and AlexNET-DFA. In brief, we observe up to +10% increase in accuracy and 2x to 4x faster learning compared with standard TM. We further employ the Convolutional TM to document interpretable results on the CIFAR datasets, visualizing how the heatmaps produced by the TM become more interpretable with drop clause. We also evaluate how drop clause affects learning robustness by introducing corruptions and alterations in the image/language test data. Our results show that drop clause makes TM more robust towards such changes.