# Attention Manifolds: Steering or Blocking Language Models by Editing Learned B-Spline Surfaces

arXiv: https://arxiv.org/abs/2610.00257 · HTML: https://mlsanity.com/paper/2610.00257/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 16 of its 17 references that are in the map.
- First posted on arXiv 24 Sep 2026.

## Builds on (most cited first, 16 of 16)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [RoFormer: Enhanced Transformer with Rotary Position Embedding](https://mlsanity.com/paper/2104.09864/) (2021)
- [Pointer Sentinel Mixture Models](https://mlsanity.com/paper/1609.07843/) (2016)
- [Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention](https://mlsanity.com/paper/2006.16236/) (2020)
- [GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints](https://mlsanity.com/paper/2305.13245/) (2023)
- [Representation Engineering: A Top-Down Approach to AI Transparency](https://mlsanity.com/paper/2310.01405/) (2023)
- [Steering Llama 2 via Contrastive Activation Addition](https://mlsanity.com/paper/2312.06681/) (2023)
- [KAN: Kolmogorov-Arnold Networks](https://mlsanity.com/paper/2404.19756/) (2024)
- [Gated Attention for Large Language Models: Non-linearity, Sparsity, and Attention-Sink-Free](https://mlsanity.com/paper/2505.06708/) (2025)
- [DiSAN: Directional Self-Attention Network for RNN/CNN-Free Language Understanding](https://mlsanity.com/paper/1709.04696/) (2017)
- [Nested Attention: Semantic-aware Attention Values for Concept Personalization](https://mlsanity.com/paper/2501.01407/) (2025)
- [Kolmogorov-Arnold Attention: Is Learnable Attention Better For Vision Transformers?](https://mlsanity.com/paper/2503.10632/) (2025)
- [InKAN: B-Spline KANs via Truncated Power Form](https://mlsanity.com/paper/2609.01956/) (2026)

## Abstract (arXiv)

In standard transformer attention, a source token sends the same value vector to every receiver. The query determines how much to attend but not what to extract. This work introduces attention manifolds: learned 2D B-spline surfaces Sd(qd, kd) that modulate each value dimension based on the query-key interaction. Each surface is a tensor-product cubic B-spline initialized to zero, preserving pretrained behavior. Applied to LLaMA 3.2-1B-Instruct and 3B-Instruct, attention manifolds reduce WikiText-2 validation perplexity by 2--2.5 points with 0.3% parameter overhead. Across 112 diverse prompts, surfaces change greedy-decoded output for 69% (1B) to 83% (3B) of cases, with the strongest effects on ambiguous and polysemous inputs (94--100% change rate). The surfaces improve output quality: correcting factual errors (``the CAP theorem has three main components'' → ``it is impossible to guarantee all three''), increasing precision (``impossible to know certain properties'' → ``impossible to know both position and momentum''), and adding specificity (a generic quote → an attributed Saint Augustine citation, consistently at both scales). The learned surfaces are also mechanically editable: inverting a layer's coefficients changes greedy output for 9/10 prompts (KL~0.010), providing a geometric mechanism for model steering. Setting surface coefficients to -1 creates ``attention walls'' that block value flow through specific dimensions. In a preliminary experiment, a layer-wide wall redirects an explosive-device prompt from specific instructions to general educational content, suggesting a path toward safety-oriented manifold shaping.