mlsanityData to 14 Oct 2026

Multi-Scale Manifold Alignment for Interpreting Large Language Models: A Unified Information-Geometric Framework

arXiv 2505.20333 ↗First posted 24 May 2025cs.CL cs.AIcited by 1

Not on the map. Why?

Abstract (arXiv)

We present Multi-Scale Manifold Alignment(MSMA), an information-geometric framework that decomposes LLM representations into local, intermediate, and global manifolds and learns cross-scale mappings that preserve geometry and information. Across GPT-2, BERT, RoBERTa, and T5, we observe consistent hierarchical patterns and find that MSMA improves alignment metrics under multiple estimators (e.g., relative KL reduction and MI gains with statistical significance across seeds). Controlled interventions at different scales yield distinct and architecture-dependent effects on lexical diversity, sentence structure, and discourse coherence. While our theoretical analysis relies on idealized assumptions, the empirical results suggest that multi-objective alignment offers a practical lens for analyzing cross-scale information flow and guiding representation-level control.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 16 of its 23 references that are in the map. First posted on arXiv 24 May 2025.

Builds on

16 of 23 references.

References · 16

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  4. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  5. 2021On the Opportunities and Risks of Foundation Modelscited by 4,033
  6. 2022Locating and Editing Factual Associations in GPTcited by 2,691
  7. 2018Deep contextualized word representationscited by 2,290
  8. 2019BERT Rediscovers the Classical NLP Pipelinecited by 1,059
  9. 2019What Does BERT Look At? An Analysis of BERT's Attentioncited by 1,003
  10. 2019Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Prunedcited by 856
  11. 2019Are Sixteen Heads Really Better than One?cited by 816
  12. 2020A Primer in BERTology: What we know about how BERT workscited by 815
4 more
  1. 2019How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddingscited by 784
  2. 2019Linguistic Knowledge and Transferability of Contextual Representationscited by 383
  3. 2019A Multiscale Visualization of Attention in the Transformer Modelcited by 292
  4. 2019Visualizing and Measuring the Geometry of BERTcited by 216

Built on by

Cited by year

Citing papers per year: 2025: 1.

1 paper.

  1. 2025LLM as Attention-Informed NTM and Topic Modeling as long-input Generation: Interpretability and long-Context Capabilitycited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Beyond Linear Concepts: Discovering and Aligning Non-Linear Concept Manifolds in Large Language Models
  2. 2024Diff-eRank: A Novel Rank-Based Metric for Evaluating Large Language Models
  3. 2025ENIGMA: The Geometry of Reasoning and Alignment in Large-Language Models
  4. 2025How Language Directions Align with Token Geometry in Multilingual LLMs
  5. 2026Geometric Properties of the Voronoi Tessellation in Latent Semantic Manifolds of Large Language Models
  6. 2025Steering into New Embedding Spaces: Analyzing Cross-Lingual Alignment Induced by Model Interventions in Multilingual Language Models
  7. 2026The Alignment Illusion in Multimodal Large Language Models
  8. 2026Cross-Lingual Alignment Without Joint Training: Do Monolingual Language Models Converge on Universal Representations?