mlsanityData to 7 Oct 2026

PRISM: A Geometric Risk Bound for Decomposing Drift into Scale, Shape, and Head

arXiv 2605.11608 ↗First posted 12 May 2026cs.CL cs.AI cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

A single base LLM now comes with dozens of post-training variants, quantized, LoRA-adapted, or distilled, and each has to be checked before release. Existing evaluations provide only a partial picture: benchmark scores and likelihood screens say that a variant has degraded, similarity scores such as CKA and SVCCA say how its features moved, and nothing connects the two. We connect them with one structural fact and one design choice: the prediction head is linear, so feature geometry reaches the loss, and we compare the two feature sets through an orthogonal map, which leaves the geometry being measured unchanged. From these we derive PRISM, a closed-form upper bound on the cross-entropy risk gap between a target model and a proxy variant, and prove that it splits exactly into three measurable axes: scale, shape, and head. Each axis names a failure mode and where to intervene: low-bit quantization distorts shape and, at the lowest bit-widths, also inflates activation scale; quantizing the output embedding inflates the head term, which dominates the bound at high bit-widths. Because the shape term is differentiable, the same geometry becomes a regularizer that curbs catastrophic forgetting more than experience replay. Across two model families and five benchmarks, PRISM ranks quantized and fine-tuned variants from a single forward pass at mean Spearman above 0.8, and eight reference sequences already recover the ranking that the risk gap itself needs over a hundred to reach.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 28 of its 33 references that are in the map. First posted on arXiv 12 May 2026.

Builds on

28 of 33 references.

References · 28

  1. 2024The Llama 3 Herd of Modelscited by 14,192
  2. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  3. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  4. 2025Qwen3 Technical Reportcited by 8,195
  5. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  6. 2020Scaling Laws for Neural Language Modelscited by 6,411
  7. 2016Overcoming catastrophic forgetting in neural networkscited by 4,718
  8. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  9. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  10. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  11. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
  12. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,895
16 more
  1. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,838
  2. 2019Similarity of Neural Network Representations Revisitedcited by 1,695
  3. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,338
  4. 2022LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scalecited by 881
  5. 2021BBQ: A Hand-Built Bias Benchmark for Question Answeringcited by 683
  6. 2023The Linear Representation Hypothesis and the Geometry of Large Language Modelscited by 679
  7. 2024tinyBenchmarks: evaluating LLMs with fewer examplescited by 295
  8. 2018Explicit Inductive Bias for Transfer Learning with Convolutional Networkscited by 226
  9. 2022Relative representations enable zero-shot latent space communicationcited by 175
  10. 2023Similarity of Neural Network Models: A Survey of Functional and Representational Measurescited by 128
  11. 2021Generalized Shape Metrics on Neural Representationscited by 109
  12. 2026Ministral 3cited by 88
  13. 2022Reliability of CKA as a Similarity Measure in Deep Learningcited by 82
  14. 2023Efficient Benchmarking of Language Modelscited by 71
  15. 2021Grounding Representation Similarity with Statistical Testingcited by 28
  16. 2024What Representational Similarity Measures Imply about Decodable Informationcited by 18

Built on by

Not cited yet.