# LaTIM: Measuring Latent Token-to-Token Interactions in Mamba Models

arXiv: https://arxiv.org/abs/2502.15612 · HTML: https://mlsanity.com/paper/2502.15612/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Placed in Core SSM architectures for language, one of 10 directions in State-space sequence models (S4, Mamba).
- Builds on 30 of its 44 references that are in the map.
- First posted on arXiv 21 Feb 2025.

## Placed in

- State-space sequence models (S4, Mamba) / [Core SSM architectures for language](https://mlsanity.com/ssm/core-ssm-architectures-for-language/)

## Builds on (most cited first, 30 of 30)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [Mamba: Linear-Time Sequence Modeling with Selective State Spaces](https://mlsanity.com/paper/2312.00752/) (2023)
- [Group Normalization](https://mlsanity.com/paper/1803.08494/) (2018)
- [RULER: What's the Real Context Size of Your Long-Context Language Models?](https://mlsanity.com/paper/2404.06654/) (2024)
- [Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality](https://mlsanity.com/paper/2405.21060/) (2024)
- [The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale](https://mlsanity.com/paper/2406.17557/) (2024)
- [COMET: A Neural Framework for MT Evaluation](https://mlsanity.com/paper/2009.09025/) (2020)
- [Attention is not Explanation](https://mlsanity.com/paper/1902.10186/) (2019)
- [Quantifying Attention Flow in Transformers](https://mlsanity.com/paper/2005.00928/) (2020)
- [MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategies](https://mlsanity.com/paper/2404.06395/) (2024)
- [Attention is not not Explanation](https://mlsanity.com/paper/1908.04626/) (2019)
- [HiPPO: Recurrent Memory with Optimal Polynomial Projections](https://mlsanity.com/paper/2008.07669/) (2020)
- [Parallelizing Linear Transformers with the Delta Rule over Sequence Length](https://mlsanity.com/paper/2406.06484/) (2024)
- [Repeat After Me: Transformers are Better than State Space Models at Copying](https://mlsanity.com/paper/2402.01032/) (2024)
- [Zoology: Measuring and Improving Recall in Efficient Language Models](https://mlsanity.com/paper/2312.04927/) (2023)
- [Word Alignment by Fine-tuning Embeddings on Parallel Corpora](https://mlsanity.com/paper/2101.08231/) (2021)
- [The elephant in the interpretability room: Why use attention as explanation when we have saliency methods?](https://mlsanity.com/paper/2010.05607/) (2020)
- [Hymba: A Hybrid-head Architecture for Small Language Models](https://mlsanity.com/paper/2411.13676/) (2024)
- [The Hidden Attention of Mamba Models](https://mlsanity.com/paper/2403.01590/) (2024)
- [Transformers to SSMs: Distilling Quadratic Knowledge to Subquadratic Models](https://mlsanity.com/paper/2408.10189/) (2024)
- ["Will You Find These Shortcuts?" A Protocol for Evaluating the Faithfulness of Input Salience Methods for Text Classification](https://mlsanity.com/paper/2111.07367/) (2021)
- [Measuring the Mixing of Contextual Information in the Transformer](https://mlsanity.com/paper/2203.04212/) (2022)
- [How do Decisions Emerge across Layers in Neural Models? Interpretation with Differentiable Masking](https://mlsanity.com/paper/2004.14992/) (2020)
- [Incorporating Residual and Normalization Layers into Analysis of Masked Language Models](https://mlsanity.com/paper/2109.07152/) (2021)
- [Explaining How Transformers Use Context to Build Predictions](https://mlsanity.com/paper/2305.12535/) (2023)
- [MambaLRP: Explaining Selective State Space Sequence Models](https://mlsanity.com/paper/2406.07592/) (2024)
- [The Eval4NLP Shared Task on Explainable Quality Estimation: Overview and Results](https://mlsanity.com/paper/2110.04392/) (2021)
- [Mimetic Initialization Helps State Space Models Learn to Recall](https://mlsanity.com/paper/2410.11135/) (2024)
- [How Effective are State Space Models for Machine Translation?](https://mlsanity.com/paper/2407.05489/) (2024)

## Built on by (most cited first, 6 of 6)

- [WriteSAE: Sparse Autoencoders for Recurrent State](https://mlsanity.com/paper/2605.12770/) (2026)
- [Evolving and Executing Research Plans via Double-Loop Multi-Agent Collaboration](https://mlsanity.com/paper/2510.06761/) (2025)
- [A Circuit, Not The Circuit: Non-Unique Causal Localisation of the Mamba-2 State Sink](https://mlsanity.com/paper/2606.00930/) (2026)
- [Bridging Human and Model Attention: Explainability Analysis of CNN, Mamba, and ViT Architectures with Gaze-Based Validation](https://mlsanity.com/paper/2610.12703/) (2026)
- [READ: Reading the Selective Scan for Faithful Video Attribution](https://mlsanity.com/paper/2610.13776/) (2026)
- [SuPER-Mamba: Surprisal Prioritization for Extended-Context Routing in Mamba](https://mlsanity.com/paper/2610.16636/) (2026)

## Abstract (arXiv)

State space models (SSMs), such as Mamba, have emerged as an efficient alternative to transformers for long-context sequence modeling. However, despite their growing adoption, SSMs lack the interpretability tools that have been crucial for understanding and improving attention-based architectures. While recent efforts provide insights into Mamba's internal mechanisms, they do not explicitly decompose token-wise contributions, leaving gaps in understanding how Mamba selectively processes sequences across layers. In this work, we introduce LaTIM, a novel token-level decomposition method for both Mamba-1 and Mamba-2 that enables fine-grained interpretability. We extensively evaluate our method across diverse tasks, including machine translation, copying, and retrieval-based generation, demonstrating its effectiveness in revealing Mamba's token-to-token interaction patterns.