# SymSeqBench: a unified framework for the generation and analysis of rule-based symbolic sequences and datasets

arXiv: https://arxiv.org/abs/2512.24977 · HTML: https://mlsanity.com/paper/2512.24977/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 39 of its 165 references that are in the map.
- First posted on arXiv 31 Dec 2025.

## Builds on (most cited first, 30 of 39)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Mamba: Linear-Time Sequence Modeling with Selective State Spaces](https://mlsanity.com/paper/2312.00752/) (2023)
- [Shortcut Learning in Deep Neural Networks](https://mlsanity.com/paper/2004.07780/) (2020)
- [SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems](https://mlsanity.com/paper/1905.00537/) (2019)
- [RULER: What's the Real Context Size of Your Long-Context Language Models?](https://mlsanity.com/paper/2404.06654/) (2024)
- [On the Properties of Neural Machine Translation: Encoder-Decoder Approaches](https://mlsanity.com/paper/1409.1259/) (2014)
- [Beyond Accuracy: Behavioral Testing of NLP models with CheckList](https://mlsanity.com/paper/2005.04118/) (2020)
- [Long Range Arena: A Benchmark for Efficient Transformers](https://mlsanity.com/paper/2011.04006/) (2020)
- [Neural Turing Machines](https://mlsanity.com/paper/1410.5401/) (2014)
- [Memory Networks](https://mlsanity.com/paper/1410.3916/) (2014)
- [AI and the Everything in the Whole Wide World Benchmark](https://mlsanity.com/paper/2111.15366/) (2021)
- [Training Spiking Neural Networks Using Lessons From Deep Learning](https://mlsanity.com/paper/2109.12894/) (2021)
- [Multimodal Deep Learning](https://mlsanity.com/paper/2301.04856/) (2023)
- [Neural Networks and the Chomsky Hierarchy](https://mlsanity.com/paper/2207.02098/) (2022)
- [Markerless tracking of user-defined features with deep learning](https://mlsanity.com/paper/1804.03142/) (2018)
- [SpikingJelly: An open-source machine learning infrastructure platform for spike-based intelligence](https://mlsanity.com/paper/2310.16620/) (2023)
- [The Heidelberg spiking datasets for the systematic evaluation of spiking neural networks](https://mlsanity.com/paper/1910.07407/) (2019)
- [In-Context Language Learning: Architectures and Algorithms](https://mlsanity.com/paper/2401.12973/) (2024)
- [A Geometric View of Data Complexity: Efficient Local Intrinsic Dimension Estimation with Diffusion Models](https://mlsanity.com/paper/2406.03537/) (2024)
- [SuperAnimal pretrained pose estimation models for behavioral analysis](https://mlsanity.com/paper/2203.07436/) (2022)
- [Criticality in Formal Languages and Statistical Physics](https://mlsanity.com/paper/1606.06737/) (2016)
- [Theory of Deep Learning III: explaining the non-overfitting puzzle](https://mlsanity.com/paper/1801.00173/) (2017)
- [A comparison of LSTM and GRU networks for learning symbolic sequences](https://mlsanity.com/paper/2107.02248/) (2021)
- [Line Goes Up? Inherent Limitations of Benchmarks for Evaluating Large Language Models](https://mlsanity.com/paper/2502.14318/) (2025)
- [Advancing Spatio-Temporal Processing in Spiking Neural Networks through Adaptation](https://mlsanity.com/paper/2408.07517/) (2024)
- [Spiking Neural Networks for event-based action recognition: A new task to understand their advantage](https://mlsanity.com/paper/2209.14915/) (2022)
- [Metastable dynamics of neural circuits and networks](https://mlsanity.com/paper/2110.03025/) (2021)
- [Sequence learning, prediction, and replay in networks of spiking neurons](https://mlsanity.com/paper/2111.03456/) (2021)
- [Re-run, Repeat, Reproduce, Reuse, Replicate: Transforming Code into Scientific Contributions](https://mlsanity.com/paper/1708.08205/) (2017)
- [Review of Machine-Learning Methods for RNA Secondary Structure Prediction](https://mlsanity.com/paper/2009.08868/) (2020)

## Abstract (arXiv)

Sequential structure is a key feature of multiple domains of natural cognition and behavior, such as language, movement and decision-making. Likewise, it is also a central property of tasks to which we would like to apply artificial intelligence. It is therefore of great importance to develop frameworks that allow us to evaluate sequence learning and processing in a domain agnostic fashion, whilst simultaneously providing a link to formal theories of computation and computability. To address this need, we introduce two complementary software tools: SymSeq, designed to rigorously generate and analyze structured symbolic sequences, and SeqBench, a comprehensive benchmark suite of rule-based sequence processing tasks to evaluate the performance of artificial learning systems in cognitively relevant domains. In combination, SymSeqBench offers versatility in investigating sequential structure across diverse knowledge domains, including experimental psycholinguistics, cognitive psychology, behavioral analysis, neuromorphic computing and artificial intelligence. Due to its basis in Formal Language Theory (FLT), SymSeqBench provides researchers in multiple domains with a convenient and practical way to apply the concepts of FLT to conceptualize and standardize their experiments, thus advancing our understanding of cognition and behavior through shared computational frameworks and formalisms. The tool is modular, openly available and accessible to the research community.