# Benchmarking the Computational and Representational Efficiency of State Space Models against Transformers on Long-Context Dyadic Sessions

arXiv: https://arxiv.org/abs/2601.01237 · HTML: https://mlsanity.com/paper/2601.01237/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 15 of its 18 references that are in the map.
- First posted on arXiv 3 Jan 2026.

## Builds on (most cited first, 15 of 15)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Mamba: Linear-Time Sequence Modeling with Selective State Spaces](https://mlsanity.com/paper/2312.00752/) (2023)
- [RoFormer: Enhanced Transformer with Rotary Position Embedding](https://mlsanity.com/paper/2104.09864/) (2021)
- [FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness](https://mlsanity.com/paper/2205.14135/) (2022)
- [Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality](https://mlsanity.com/paper/2405.21060/) (2024)
- [Repeat After Me: Transformers are Better than State Space Models at Copying](https://mlsanity.com/paper/2402.01032/) (2024)
- [Automated Quality Assessment of Cognitive Behavioral Therapy Sessions Through Highly Contextualized Language Representations](https://mlsanity.com/paper/2102.11573/) (2021)
- [COMPASS: Computational Mapping of Patient-Therapist Alliance Strategies with Language Modeling](https://mlsanity.com/paper/2402.14701/) (2024)
- [Contextual Representation Flow Across Depth in State-Space and Transformer Architectures](https://mlsanity.com/paper/2510.06640/) (2025)
- [Characterizing State Space Model and Hybrid Language Model Performance with Long Context](https://mlsanity.com/paper/2507.12442/) (2025)

## Abstract (arXiv)

State Space Models (SSMs) have emerged as a promising alternative to Transformers for long-context sequence modeling, offering linear O(N) computational complexity compared to the Transformer's quadratic O(N2) scaling. This paper presents a comprehensive benchmarking study comparing the Mamba SSM against the LLaMA Transformer on long-context sequences, using dyadic therapy sessions as a representative test case. We evaluate both architectures across two dimensions: (1) computational efficiency, where we measure memory usage and inference speed from 512 to 8,192 tokens, and (2) representational efficiency, where we analyze hidden state dynamics and attention patterns. Our findings provide actionable insights for practitioners working with long-context applications, establishing precise conditions under which SSMs offer advantages over Transformers.