# How Transformers Reject Wrong Answers: Rotational Dynamics of Factual Constraint Processing arXiv: https://arxiv.org/abs/2603.13259 ยท HTML: https://mlsanity.com/paper/2603.13259/ ## Key facts - Cited by 1 papers in the map (data through 14 Oct 2026). - Placed in Probing deception, bias and beliefs, one of 13 directions in Mechanistic interpretability. - Builds on 33 of its 46 references that are in the map. - First posted on arXiv 25 Feb 2026. ## Placed in - Mechanistic interpretability / [Probing deception, bias and beliefs](https://mlsanity.com/mechinterp/probing-deception-bias-and-beliefs/) ## Builds on (most cited first, 30 of 33) - [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019) - [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024) - [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023) - [Mistral 7B](https://mlsanity.com/paper/2310.06825/) (2023) - [Locating and Editing Factual Associations in GPT](https://mlsanity.com/paper/2202.05262/) (2022) - [Emergent Abilities of Large Language Models](https://mlsanity.com/paper/2206.07682/) (2022) - [Qwen2 Technical Report](https://mlsanity.com/paper/2407.10671/) (2024) - [Survey of Hallucination in Natural Language Generation](https://mlsanity.com/paper/2202.03629/) (2022) - [Gemma 2: Improving Open Language Models at a Practical Size](https://mlsanity.com/paper/2408.00118/) (2024) - [A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions](https://mlsanity.com/paper/2311.05232/) (2023) - [Understanding intermediate layers using linear classifier probes](https://mlsanity.com/paper/1610.01644/) (2016) - [Transformer Feed-Forward Layers Are Key-Value Memories](https://mlsanity.com/paper/2012.14913/) (2020) - [Representation Engineering: A Top-Down Approach to AI Transparency](https://mlsanity.com/paper/2310.01405/) (2023) - [Inference-Time Intervention: Eliciting Truthful Answers from a Language Model](https://mlsanity.com/paper/2306.03341/) (2023) - [Geometric Deep Learning: Grids, Groups, Graphs, Geodesics, and Gauges](https://mlsanity.com/paper/2104.13478/) (2021) - [On Faithfulness and Factuality in Abstractive Summarization](https://mlsanity.com/paper/2005.00661/) (2020) - [Discovering Latent Knowledge in Language Models Without Supervision](https://mlsanity.com/paper/2212.03827/) (2022) - [The Linear Representation Hypothesis and the Geometry of Large Language Models](https://mlsanity.com/paper/2311.03658/) (2023) - [The Geometry of Truth: Emergent Linear Structure in Large Language Model Representations of True/False Datasets](https://mlsanity.com/paper/2310.06824/) (2023) - [The Internal State of an LLM Knows When It's Lying](https://mlsanity.com/paper/2304.13734/) (2023) - [Designing and Interpreting Probes with Control Tasks](https://mlsanity.com/paper/1909.03368/) (2019) - [Eliciting Latent Predictions from Transformers with the Tuned Lens](https://mlsanity.com/paper/2303.08112/) (2023) - [Knowledge Neurons in Pretrained Transformers](https://mlsanity.com/paper/2104.08696/) (2021) - [Dissecting Recall of Factual Associations in Auto-Regressive Language Models](https://mlsanity.com/paper/2304.14767/) (2023) - [Are Emergent Abilities of Large Language Models a Mirage?](https://mlsanity.com/paper/2304.15004/) (2023) - [Language Models Represent Space and Time](https://mlsanity.com/paper/2310.02207/) (2023) - [Does Localization Inform Editing? Surprising Differences in Causality-Based Localization vs. Knowledge Editing in Language Models](https://mlsanity.com/paper/2301.04213/) (2023) - [Linearity of Relation Decoding in Transformer Language Models](https://mlsanity.com/paper/2308.09124/) (2023) - [Analyzing Transformers in Embedding Space](https://mlsanity.com/paper/2209.02535/) (2022) - [Evaluating the Factual Consistency of Large Language Models Through News Summarization](https://mlsanity.com/paper/2211.08412/) (2022) ## Built on by (most cited first, 1 of 1) - [The Anatomy of a Truth Direction: Knowledge-Dependent Dimensionality, a Relational Law, and a Shared Category Geometry in Small Language Models](https://mlsanity.com/paper/2607.16741/) (2026) ## Abstract (arXiv) When a decoder-only transformer is forced to process matched correct and incorrect single-token continuations of a factual query, the two pathways through hidden-state space diverge: displacement vectors from the query-only representation keep near-equal magnitude but rotate apart, with angular separation growing through mid-depth before late layers resolve an asymmetric outcome. A logit-lens preference in the incorrect run falls far below the equal-probability prior (roughly 11.5x more mass on the incorrect token than the correct one). We read this pattern, rotational divergence then late-layer asymmetric commitment, as the geometric signature of the model externally appearing to reject a wrong continuation, while staying explicit that it is observational, not causal: the incorrect run could equally reflect the model conforming to the token it is forced to carry, which only a random-token control can settle. It holds across six decoder-only transformers spanning four architecture families (Llama, Mistral, Gemma, StableLM) from 1B to 13B parameters; a seventh (Qwen2 1.5B) is flat under our protocol, plausibly a tokenizer artefact, leaving an emergence threshold open. Linear probes recover the distinction at intermediate depth, and cross-domain transfer is structurally asymmetric, a financial-medical corridor transferring far better than transport pairs. Where single-layer activation patching is cleanly interpretable (LLaMA-2 13B, Mistral 7B) it yields no layer band of consistent recovery; a third model (StableLM-2 1.6B) recovers uniformly above the ceiling, which we diagnose as a code-path artefact and exclude. Under this scoped null the late-layer asymmetry is not localized to a single component, fitting a distributed-by-trajectory account rather than single-layer localized recall. We document this with forced-completion probing across seven models, three domains, and 300 queries.