# SoPa: Bridging CNNs, RNNs, and Weighted Finite-State Machines

arXiv: https://arxiv.org/abs/1805.06061 · HTML: https://mlsanity.com/paper/1805.06061/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 15 May 2018.

## Built on by (most cited first, 8 of 8)

- [Towards Faithful Model Explanation in NLP: A Survey](https://mlsanity.com/paper/2209.11326/) (2022)
- [Tighter Bounds on the Expressivity of Transformer Encoders](https://mlsanity.com/paper/2301.10743/) (2023)
- [Neuro-Symbolic Language Modeling with Automaton-augmented Retrieval](https://mlsanity.com/paper/2201.12431/) (2022)
- [Formal Language Theory Meets Modern NLP](https://mlsanity.com/paper/2102.10094/) (2021)
- [The Surprising Computational Power of Nondeterministic Stack RNNs](https://mlsanity.com/paper/2210.01343/) (2022)
- [Algorithms for Acyclic Weighted Finite-State Automata with Failure Arcs](https://mlsanity.com/paper/2301.06862/) (2023)
- [Interpretable Predictive Models for Healthcare via Rational Logistic Regression](https://mlsanity.com/paper/2411.03224/) (2024)
- [Higher-order Derivatives of Weighted Finite-state Machines](https://mlsanity.com/paper/2106.00749/) (2021)

## Abstract (arXiv)

Recurrent and convolutional neural networks comprise two distinct families of models that have proven to be useful for encoding natural language utterances. In this paper we present SoPa, a new model that aims to bridge these two approaches. SoPa combines neural representation learning with weighted finite-state automata (WFSAs) to learn a soft version of traditional surface patterns. We show that SoPa is an extension of a one-layer CNN, and that such CNNs are equivalent to a restricted version of SoPa, and accordingly, to a restricted form of WFSA. Empirically, on three text classification tasks, SoPa is comparable or better than both a BiLSTM (RNN) baseline and a CNN baseline, and is particularly useful in small data settings.