MANAR: Memory-augmented Attention with Navigational Abstract Conceptual Representation
Not on the map. Why?
Abstract (arXiv)
MANAR (Memory-augmented Attention with Navigational Abstract Conceptual Representation), contextualization layer generalizes standard multi-head attention (MHA) by instantiating the principles of Global Workspace Theory (GWT). While MHA enables unconstrained all-to-all communication, it lacks the functional bottleneck and global integration mechanisms hypothesized in cognitive models of consciousness. MANAR addresses this by implementing a central workspace through a trainable memory of abstract concepts and an Abstract Conceptual Representation (ACR). The architecture follows a two-stage logic that maps directly to GWT mechanics: (i) an integration phase, where retrieved memory concepts converge to form a collective "mental image" (the ACR) based on input stimuli; and (ii) a broadcasting phase, where this global state navigates and informs the contextualization of individual local tokens. We demonstrate that efficient linear-time scaling is a fundamental architectural byproduct of instantiating GWT functional bottleneck, as routing global information through a constant-sized ACR resolves the quadratic complexity inherent in standard attention. MANAR is a compatible re-parameterization of MHA with identical semantic roles for its projections, enabling knowledge transfer from pretrained transformers via weight-copy and thus overcoming the adoption barriers of structurally incompatible linear-time alternatives. MANAR enables non-convex contextualization, synthesizing representations that provably lie outside the convex hull of input tokens - a mathematical reflection of the creative synthesis described in GWT. Empirical evaluations confirm that MANAR matches or exceeds strong baselines across language (GLUE score of 85.1), vision (83.9% ImageNet-1K), and speech (2.7% WER on LibriSpeech), positioning it as an efficient and expressive alternative to quadratic attention.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 43 of its 57 references that are in the map. First posted on arXiv 19 Mar 2026.
Builds on
43 of 57 references.
References · 43
- 2017Attention Is All You Needcited by 60,361
- 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
- 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
- 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
- 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
- 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,718
- 2020Training data-efficient image transformers & distillation through attentioncited by 4,586
- 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,905
- 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,824
- 2017Billion-scale similarity search with GPUscited by 2,497
- 2021Efficiently Modeling Long Sequences with Structured State Spacescited by 2,465
- 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,342
31 more
- 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,850
- 2017Decoupled Weight Decay Regularizationcited by 1,831
- 2021ViViT: A Video Vision Transformercited by 1,620
- 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,386
- 2024Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Dualitycited by 1,336
- 2024Vision Mamba: Efficient Visual Representation Learning with Bidirectional State Space Modelcited by 1,056
- 2022data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Languagecited by 727
- 2019Compressive Transformers for Long-Range Sequence Modellingcited by 726
- 2024KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cachecited by 637
- 2024QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMscited by 634
- 2023Retentive Network: A Successor to Transformer for Large Language Modelscited by 613
- 2024Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder for Fast, Memory Efficient, and Long Context Finetuning and Inferencecited by 554
- 2024SpinQuant: LLM quantization with learned rotationscited by 471
- 2024Titans: Learning to Memorize at Test Timecited by 328
- 2024InfiniGen: Efficient Generative Inference of Large Language Models with Dynamic KV Cache Managementcited by 245
- 2022Memorizing Transformerscited by 226
- 2023Landmark Attention: Random-Access Infinite Context Length for Transformerscited by 213
- 2024InfLLM: Training-Free Long-Context Extrapolation for LLMs with an Efficient Context Memorycited by 197
- 2024ShadowKV: KV Cache in Shadows for High-Throughput Long-Context LLM Inferencecited by 134
- 2019Large Memory Layers with Product Keyscited by 124
- 2020Memory Transformercited by 76
- 2025ATLAS: Learning to Optimally Memorize the Context at Test Timecited by 66
- 2024Human-inspired Episodic Memory for Infinite Context LLMscited by 62
- 2025Seq vs Seq: An Open Suite of Paired Encoders and Decoderscited by 38
- 2023DinoSR: Self-Distillation and Online Clustering for Self-supervised Speech Representation Learningcited by 37
- 2023MosaicBERT: A Bidirectional Encoder Optimized for Fast Pretrainingcited by 34
- 2023Object Detection with Transformers: A Reviewcited by 22
- 2021XtremeDistilTransformers: Task Transfer for Task-agnostic Distillationcited by 18
- 2024AMD: Automatic Multi-step Distillation of Large-scale Vision Modelscited by 12
- 2024MemLong: Memory-Augmented Retrieval for Long Text Modelingcited by 11
- 2025Optimizing Knowledge Distillation in Transformers: Enabling Multi-Head Attention without Alignment Barrierscited by 3
Built on by
Not cited yet.