# Characterizing Large Language Model Geometry Helps Solve Toxicity Detection and Generation

arXiv: https://arxiv.org/abs/2312.01648 · HTML: https://mlsanity.com/paper/2312.01648/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 39 of its 52 references that are in the map.
- First posted on arXiv 4 Dec 2023.

## Builds on (most cited first, 30 of 39)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [RoFormer: Enhanced Transformer with Rotary Position Embedding](https://mlsanity.com/paper/2104.09864/) (2021)
- [Mistral 7B](https://mlsanity.com/paper/2310.06825/) (2023)
- [Training Compute-Optimal Large Language Models](https://mlsanity.com/paper/2203.15556/) (2022)
- [The Pile: An 800GB Dataset of Diverse Text for Language Modeling](https://mlsanity.com/paper/2101.00027/) (2020)
- [Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 small](https://mlsanity.com/paper/2211.00593/) (2022)
- [Probing Classifiers: Promises, Shortcomings, and Advances](https://mlsanity.com/paper/2102.12452/) (2021)
- [Discovering Latent Knowledge in Language Models Without Supervision](https://mlsanity.com/paper/2212.03827/) (2022)
- [Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuning](https://mlsanity.com/paper/2012.13255/) (2020)
- [ToxiGen: A Large-Scale Machine-Generated Dataset for Adversarial and Implicit Hate Speech Detection](https://mlsanity.com/paper/2203.09509/) (2022)
- [Attention is Not All You Need: Pure Attention Loses Rank Doubly Exponentially with Depth](https://mlsanity.com/paper/2103.03404/) (2021)
- [HateXplain: A Benchmark Dataset for Explainable Hate Speech Detection](https://mlsanity.com/paper/2012.10289/) (2020)
- [Understanding Dimensional Collapse in Contrastive Self-supervised Learning](https://mlsanity.com/paper/2110.09348/) (2021)
- [Explainability for Large Language Models: A Survey](https://mlsanity.com/paper/2309.01029/) (2023)
- [The Intrinsic Dimension of Images and Its Impact on Learning](https://mlsanity.com/paper/2104.08894/) (2021)
- [The Lottery Ticket Hypothesis for Pre-trained BERT Networks](https://mlsanity.com/paper/2007.12223/) (2020)
- [HateBERT: Retraining BERT for Abusive Language Detection in English](https://mlsanity.com/paper/2010.12472/) (2020)
- [Better Fine-Tuning by Reducing Representational Collapse](https://mlsanity.com/paper/2008.03156/) (2020)
- [Probing the Probing Paradigm: Does Probing Accuracy Entail Task Relevance?](https://mlsanity.com/paper/2005.00719/) (2020)
- [A Toy Model of Universality: Reverse Engineering How Networks Learn Group Operations](https://mlsanity.com/paper/2302.03025/) (2023)
- [Signal Propagation in Transformers: Theoretical Perspectives and the Role of Rank Collapse](https://mlsanity.com/paper/2206.03126/) (2022)
- [Analyzing Transformers in Embedding Space](https://mlsanity.com/paper/2209.02535/) (2022)
- [Challenges in Automated Debiasing for Toxic Language Detection](https://mlsanity.com/paper/2102.00086/) (2021)
- [Challenges for Toxic Comment Classification: An In-Depth Error Analysis](https://mlsanity.com/paper/1809.07572/) (2018)
- [Mad Max: Affine Spline Insights into Deep Learning](https://mlsanity.com/paper/1805.06576/) (2018)
- [Mimetic Initialization of Self-Attention Layers](https://mlsanity.com/paper/2305.09828/) (2023)

## Built on by (most cited first, 12 of 12)

- [Large Language Model Safety: A Holistic Survey](https://mlsanity.com/paper/2412.17686/) (2024)
- [Position: Beyond Euclidean -- Foundation Models Should Embrace Non-Euclidean Geometries](https://mlsanity.com/paper/2504.08896/) (2025)
- [Geometric Signatures of Compositionality Across a Language Model's Lifetime](https://mlsanity.com/paper/2410.01444/) (2024)
- [A Survey on Efficient Large Language Model Training: From Data-centric Perspectives](https://mlsanity.com/paper/2510.25817/) (2025)
- [Reasoning in Large Language Models: A Geometric Perspective](https://mlsanity.com/paper/2407.02678/) (2024)
- [Tracing the complexity profiles of different linguistic phenomena through the intrinsic dimension of LLM representations](https://mlsanity.com/paper/2601.03779/) (2026)
- [NerVE: Nonlinear Eigenspectrum Dynamics in LLM Feed-Forward Networks](https://mlsanity.com/paper/2603.06922/) (2026)
- [The Linear Centroids Hypothesis: Features as Directions Learned by Local Experts](https://mlsanity.com/paper/2604.11962/) (2026)
- [Confidence is Not Competence](https://mlsanity.com/paper/2510.24772/) (2025)
- [Characterizing the Discrete Geometry of ReLU Networks](https://mlsanity.com/paper/2606.07728/) (2026)
- [Emergent Causal-Geometric Dynamics Across Depth in Large Language Models](https://mlsanity.com/paper/2602.04931/) (2026)
- [Geometry-Aware Graph Construction via Adaptive Spectral Bandwidth Control](https://mlsanity.com/paper/2609.03306/) (2026)

## Abstract (arXiv)

Large Language Models (LLMs) drive current AI breakthroughs despite very little being known about their internal representations. In this work, we propose to shed the light on LLMs inner mechanisms through the lens of geometry. In particular, we develop in closed form (i) the intrinsic dimension in which the Multi-Head Attention embeddings are constrained to exist and (ii) the partition and per-region affine mappings of the feedforward (MLP) network of LLMs' layers. Our theoretical findings further enable the design of novel principled solutions applicable to state-of-the-art LLMs. First, we show that, through our geometric understanding, we can bypass LLMs' RLHF protection by controlling the embedding's intrinsic dimension through informed prompt manipulation. Second, we derive interpretable geometrical features that can be extracted from any (pre-trained) LLM, providing a rich abstract representation of their inputs. We observe that these features are sufficient to help solve toxicity detection, and even allow the identification of various types of toxicity. Our results demonstrate how, even in large-scale regimes, exact theoretical results can answer practical questions in LLMs. Code: https://github.com/RandallBalestriero/SplineLLM