mlsanityData to 13 Oct 2026

GeoLAN: Geometric Learning of Latent Explanatory Directions in Large Language Models

arXiv 2603.19460 ↗First posted 19 Mar 2026cs.LG cs.CGcited by 0

Not on the map. Why?

Abstract (arXiv)

Large language models (LLMs) demonstrate strong performance, but they often lack transparency. We introduce GeoLAN, a training framework that treats token representations as geometric trajectories and applies stickiness conditions inspired by recent developments related to the Kakeya Conjecture. We have developed two differentiable regularizers, Katz-Tao Convex Wolff (KT-CW) and Katz-Tao Attention (KT-Attn), that promote isotropy and encourage diverse attention. Our experiments with Gemma-3 (1B, 4B, 12B) and Llama-3-8B show that GeoLAN frequently maintains task accuracy while improving geometric metrics and reducing certain fairness biases. These benefits are most significant in mid-sized models. Our findings reveal scale-dependent trade-offs between geometric precision and performance, suggesting that geometry-aware training is a promising approach to enhance mechanistic interpretability.

Key facts. No citing papers in the map yet (data through 13 Oct 2026). Builds on 19 of its 32 references that are in the map. First posted on arXiv 19 Mar 2026.

Builds on

19 of 32 references.

References · 19

  1. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,549
  2. 2020Measuring Massive Multitask Language Understandingcited by 7,876
  3. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,949
  4. 2023Sparse Autoencoders Find Highly Interpretable Features in Language Modelscited by 1,391
  5. 2018A Survey Of Methods For Explaining Black Box Modelscited by 971
  6. 2023Progress measures for grokking via mechanistic interpretabilitycited by 831
  7. 2019How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddingscited by 774
  8. 2020Transformer Interpretability Beyond Attention Visualizationcited by 623
  9. 2024LLMs Know More Than They Show: On the Intrinsic Representation of LLM Hallucinationscited by 236
  10. 2021Post-hoc Interpretability for Neural NLP: A Surveycited by 156
  11. 2022Towards Faithful Model Explanation in NLP: A Surveycited by 129
  12. 2021IsoScore: Measuring the Uniformity of Embedding Space Utilizationcited by 52
7 more
  1. 2025The Hidden Dimensions of LLM Alignment: A Multi-Dimensional Analysis of Orthogonal Safety Directionscited by 40
  2. 2025Tracing the Representation Geometry of Language Models from Pretraining to Post-trainingcited by 35
  3. 2025The Rogue Scalpel: Activation Steering Compromises LLM Safetycited by 21
  4. 2025From superposition to sparse codes: interpretable representations in neural networkscited by 16
  5. 2025Unboxing the Black Box: Mechanistic Interpretability for Algorithmic Understanding of Neural Networkscited by 6
  6. 2025Combatting Dimensional Collapse in LLM Pre-Training Data via Diversified File Selectioncited by 4
  7. 2025Mechanistic Interpretability of Code Correctness in LLMs via Sparse Autoencoderscited by 4

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026A Geometric Perspective on Next-Token Prediction in Large Language Models: Three Emerging Phases
  2. 2025Do Large Language Models Truly Understand Geometric Structures?
  3. 2026GeoAlign: Geometric Feature Realignment for MLLM Spatial Reasoning
  4. 2026Measuring Representation Robustness in Large Language Models for Geometry
  5. 2026GeoLatent: Geometry-Guided Latent Structuring with Routed Optimization for 3D Reasoning
  6. 2025Shared Global and Local Geometry of Language Model Embeddings
  7. 2023G-LLaVA: Solving Geometric Problem with Multi-Modal Large Language Model
  8. 2025HELM: Hyperbolic Large Language Models via Mixture-of-Curvature Experts