mlsanityData to 7 Oct 2026

Coordinates of Capability: A Unified MTMM-Geometric Framework for LLM Evaluation

arXiv 2605.08522 ↗First posted 8 May 2026cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

The evaluation of Large Language Models (LLMs) faces a critical challenge in construct validity, where fragmented benchmarks and ad hoc metrics frequently conflate method variance, such as prompt sensitivity, with true latent capabilities. Concurrently, emerging research suggests that LLM capabilities and outputs can be modeled as continuous geometric manifolds. In this Systematization of Knowledge (SoK), we bridge these paradigms by proposing a generalized Multi-Trait Multi-Method (MTMM) framework for LLM evaluation. We formalize and unify nine evaluation metrics, including Paraphrase Instability, Drift Score, Overton Width, and Pluralism Score, interpreting them not as isolated scalar values but as geometric measurements within a shared latent coordinate space. This spatial unification factorizes model behavior into three orthogonal latent dimensions: (1) Instability and Sensitivity, (2) Position and Alignment, and (3) Coverage and Expressiveness. By systematically separating task-irrelevant perturbations from true capability spans, the framework provides a theoretically grounded and domain-agnostic taxonomy for robust and empirically stable benchmark design.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 19 of its 26 references that are in the map. First posted on arXiv 8 May 2026.

Builds on

19 of 26 references.

References · 19

  1. 2024Justice or Prejudice? Quantifying Biases in LLM-as-a-Judgecited by 354
  2. 2024Political Compass or Spinning Arrow? Towards More Meaningful Evaluations for Values and Opinions in Large Language Modelscited by 162
  3. 2024Large Language Models are Inconsistent and Biased Evaluatorscited by 146
  4. 2024ProSA: Assessing and Understanding the Prompt Sensitivity of LLMscited by 145
  5. 2025MME-CoT: Benchmarking Chain-of-Thought in Large Multimodal Models for Reasoning Quality, Robustness, and Efficiencycited by 123
  6. 2025Measuring what Matters: Construct Validity in Large Language Model Benchmarkscited by 102
  7. 2023Evaluation Metrics in the Era of GPT-4: Reliably Evaluating Large Language Models on Sequence to Sequence Taskscited by 53
  8. 2024Social Bias Evaluation for Large Language Models Requires Prompt Variationscited by 39
  9. 2025Shared Global and Local Geometry of Language Model Embeddingscited by 34
  10. 2025A Survey on Large Language Model Benchmarkscited by 32
  11. 2023Sensitivity and Robustness of Large Language Models to Prompt Template in Japanese Text Classification Taskscited by 25
  12. 2025Drift No More? Context Equilibria in Multi-Turn LLM Interactionscited by 21
7 more
  1. 2024Mapping and Influencing the Political Ideology of Large Language Models using Synthetic Personascited by 16
  2. 2026BiasScope: Towards Automated Detection of Bias in LLM-as-a-Judge Evaluationcited by 12
  3. 2025Time-To-Inconsistency: A Survival Analysis of Large Language Model Robustness to Adversarial Attackscited by 6
  4. 2026Evaluating Performance Drift from Model Switching in Multi-Turn LLM Systemscited by 4
  5. 2025Visualizing LLM Latent Space Geometry Through Dimensionality Reductioncited by 3
  6. 2026Quantifying construct validity in large language model evaluationscited by 3
  7. 2025Large Language Models Can Be a Viable Substitute for Expert Political Surveys When a Shock Disrupts Traditional Measurement Approachescited by 1

Built on by

Not cited yet.