# Attention is Case-Sensitive

arXiv: https://arxiv.org/abs/2608.03711 · HTML: https://mlsanity.com/paper/2608.03711/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 24 of its 41 references that are in the map.
- First posted on arXiv 4 Aug 2026.

## Builds on (most cited first, 24 of 24)

- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [Locating and Editing Factual Associations in GPT](https://mlsanity.com/paper/2202.05262/) (2022)
- [SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing](https://mlsanity.com/paper/1808.06226/) (2018)
- [Know What You Don't Know: Unanswerable Questions for SQuAD](https://mlsanity.com/paper/1806.03822/) (2018)
- [gpt-oss-120b & gpt-oss-20b Model Card](https://mlsanity.com/paper/2508.10925/) (2025)
- [Generation and Comprehension of Unambiguous Object Descriptions](https://mlsanity.com/paper/1511.02283/) (2015)
- [On the Cross-lingual Transferability of Monolingual Representations](https://mlsanity.com/paper/1910.11856/) (2019)
- [Programming Refusal with Conditional Activation Steering](https://mlsanity.com/paper/2409.05907/) (2024)
- [Improving Instruction-Following in Language Models through Activation Steering](https://mlsanity.com/paper/2410.12877/) (2024)
- [Found in the Middle: Calibrating Positional Attention Bias Improves Long Context Utilization](https://mlsanity.com/paper/2406.16008/) (2024)
- [Tell Your Model Where to Attend: Post-hoc Attention Steering for LLMs](https://mlsanity.com/paper/2311.02262/) (2023)
- [On the Emergence of Position Bias in Transformers](https://mlsanity.com/paper/2502.01951/) (2025)
- [Structural Persistence in Language Models: Priming as a Window into Abstract Language Representations](https://mlsanity.com/paper/2109.14989/) (2021)
- [Spotlight Your Instructions: Instruction-following with Dynamic Attention Steering](https://mlsanity.com/paper/2505.12025/) (2025)
- [Understanding and Mitigating Tokenization Bias in Language Models](https://mlsanity.com/paper/2406.16829/) (2024)
- [When Language Overrules: Revealing Text Dominance in Multimodal Large Language Models](https://mlsanity.com/paper/2508.10552/) (2025)
- [Model Tells Itself Where to Attend: Faithfulness Meets Automatic Attention Steering](https://mlsanity.com/paper/2409.10790/) (2024)
- [Steering LLM Reasoning Through Bias-Only Adaptation](https://mlsanity.com/paper/2505.18706/) (2025)
- [Mass-Editing Memory with Attention in Transformers: A cross-lingual exploration of knowledge](https://mlsanity.com/paper/2502.02173/) (2025)
- [Value-State Gated Attention for Mitigating Extreme-Token Phenomena in Transformers](https://mlsanity.com/paper/2510.09017/) (2025)
- [Efficient and accurate steering of Large Language Models through attention-guided feature learning](https://mlsanity.com/paper/2602.00333/) (2026)
- [Spectral Attention Steering for Prompt Highlighting](https://mlsanity.com/paper/2603.01281/) (2026)
- [CTR-Sink: Attention Sink for Language Models in Click-Through Rate Prediction](https://mlsanity.com/paper/2508.03668/) (2025)
- [XML Prompting as Grammar-Constrained Interaction: Fixed-Point Semantics, Convergence Guarantees, and Human-AI Protocols](https://mlsanity.com/paper/2509.08182/) (2025)

## Abstract (arXiv)

In human visual perception, uppercase lettering serves as a natural salience cue that captures attention within lowercase text. In this paper, we present a systematic empirical characterization study revealing that Large Language Models (LLMs) exhibit an analogous property: letter casing modulates internal attention allocation. Through analysis across 13 models, nine LLMs and four Vision-Language Models (VLMs), with diverse tokenization schemes, we show that formatting target information in alternating or uppercase against a lowercase context concentrates attention on those textual spans. In text this effect is universal, holding across every evaluated non-reasoning model. We frame it as a previously under-explored latent property of pretrained transformers rather than a prescriptive method. Our investigation reveals a central attention-performance divergence: while this "casing effect" robustly shifts attention, its impact on downstream accuracy is non-trivial, increased concentration does not inherently improve task accuracy and, in high-entropy contexts like alternating case, can degrade it. We further identify a boundary condition: the deliberative "thinking" phase in reasoning models acts as a semantic buffer that mitigates typographic sensitivity in text. Extending the study to VLMs, we find the effect transfers partially: the same prompt-side casing reorganizes cross-modal attention along two coupled axes, predominantly a macroscopic disengagement from the image toward the text prompt, and secondarily a concentration of the residual visual attention on the target region. By isolating casing as a zero-shot mechanism for attention steering that requires no model access or fine-tuning, we provide a new foundational understanding of how pretraining internalizes typographic emphasis.