# MapFormer: Self-Supervised Learning of Cognitive Maps with Input-Dependent Positional Embeddings

arXiv: https://arxiv.org/abs/2511.19279 · HTML: https://mlsanity.com/paper/2511.19279/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 23 of its 42 references that are in the map.
- First posted on arXiv 24 Nov 2025.

## Builds on (most cited first, 23 of 23)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Mamba: Linear-Time Sequence Modeling with Selective State Spaces](https://mlsanity.com/paper/2312.00752/) (2023)
- [RoFormer: Enhanced Transformer with Rotary Position Embedding](https://mlsanity.com/paper/2104.09864/) (2021)
- [FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness](https://mlsanity.com/paper/2205.14135/) (2022)
- [Efficiently Modeling Long Sequences with Structured State Spaces](https://mlsanity.com/paper/2111.00396/) (2021)
- [Linformer: Self-Attention with Linear Complexity](https://mlsanity.com/paper/2006.04768/) (2020)
- [Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality](https://mlsanity.com/paper/2405.21060/) (2024)
- [The NarrativeQA Reading Comprehension Challenge](https://mlsanity.com/paper/1712.07040/) (2017)
- [BLiMP: The Benchmark of Linguistic Minimal Pairs for English](https://mlsanity.com/paper/1912.00582/) (2019)
- [Evaluating the World Model Implicit in a Generative Model](https://mlsanity.com/paper/2406.03689/) (2024)
- [Self-Attention Networks Can Process Bounded Hierarchical Languages](https://mlsanity.com/paper/2105.11115/) (2021)
- [Relating transformers to models and neural representations of the hippocampal formation](https://mlsanity.com/paper/2112.04035/) (2021)
- [Contextual Position Encoding: Learning to Count What's Important](https://mlsanity.com/paper/2405.18719/) (2024)
- [PaTH Attention: Position Encoding via Accumulating Householder Transformations](https://mlsanity.com/paper/2505.16381/) (2025)
- [Memory-Augmented Recurrent Neural Networks Can Learn Generalized Dyck Languages](https://mlsanity.com/paper/1911.03329/) (2019)
- [Disentanglement with Biological Constraints: A Theory of Functional Cell Types](https://mlsanity.com/paper/2210.01768/) (2022)
- [On the Out-Of-Distribution Generalization of Multimodal Large Language Models](https://mlsanity.com/paper/2402.06599/) (2024)
- [On the Practical Ability of Recurrent Neural Networks to Recognize Hierarchical Languages](https://mlsanity.com/paper/2011.03965/) (2020)
- [Actionable Neural Representations: Grid Cells from Minimal Constraints](https://mlsanity.com/paper/2209.15563/) (2022)
- [Decoupling the "What" and "Where" With Polar Coordinate Positional Embeddings](https://mlsanity.com/paper/2509.10534/) (2025)
- [Meta-Learning Neural Mechanisms rather than Bayesian Priors](https://mlsanity.com/paper/2503.16048/) (2025)
- [SPaRC: A Spatial Pathfinding Reasoning Challenge](https://mlsanity.com/paper/2505.16686/) (2025)
- [Rethinking Addressing in Language Models via Contexualized Equivariant Positional Encoding](https://mlsanity.com/paper/2501.00712/) (2025)

## Abstract (arXiv)

A cognitive map is an internal model which encodes the abstract relationships among entities in the world, giving humans and animals the flexibility to adapt to new situations, with a strong out-of-distribution (OOD) generalization that current AI systems still do not possess. To bridge this gap, we introduce textitMapFormers, new Transformer-based architectures, which can learn cognitive maps from observational data and perform path-integration without supervision. Cognitive maps are learned in the model by disentangling structural relationships in the inputs from their specific content, a property that can be achieved by updating position encodings with input-dependent matrices, built as exponentials of learned combinations of Lie-algebra generators. We developed two variants of textitMapFormers that unify absolute and relative positional encoding to model episodic (EM) and working memory (WM), respectively. We tested textitMapFormers on several formal tasks targeting distinct cognitive capacities, including gating, 2D navigation and nested hierarchies (Dyck Languages). Our results demonstrate that textitMapFormers significantly outperform current AI architectures, achieving near-perfect OOD generalization where standard models fail. Furthermore, we show that textitMapFormers are scalable; evaluations on naturalistic data yield perplexity improvements over baselines, suggesting that these principles extend to large-scale, real-world domains. These results are obtained through efficient parallel computation on commutative maps, though our models can also learn non-commutative cognitive maps via sequential path-integration. Overall, these results suggest that input-dependent matrices provide a critical structural bias, by disentangling abstract relations from content in order to drive robust OOD generalization.