mlsanityData to 14 Oct 2026

When Graph-JEPA Learns the Wrong Thing: Diagnosing and Repairing Category-Conditional Collapse

arXiv 2608.20516 ↗First posted 20 Aug 2026cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Joint-embedding predictive architectures are selected almost universally by linear probing and effective rank. We report a case where both read healthily while the representation carries zero usable instance information. We repair it, and a second failure appears: the repaired metric saturates on a target carrying no structural information. Our corpus is a scientific-reasoning graph over 57,903 articles, each a subgraph. A Graph-JEPA predicts one masked aspect from a subgraph's remaining aspects, attaining linear-probe accuracy 0.871 and effective rank 18-47, yet retrieval recovers 0.00 of 14.4 bits (MRR 1.9e-4 vs chance 1.99e-4, p=0.98). Three upper bounds on the same pool and code recover nearly everything (+14.28, +14.34, +14.22 bits), ruling out corpus, masking, pool, and metric as causes. We trace this to variance allocation - frozen inputs place 86.05% of variance on subgraph identity and 0.40% on aspect identity, while trained latents place 0.39% and 99.61%. This is a property of the objective's optimum: the degenerate solution is a global minimum of the coupled predictor/EMA-target objective, present already at init. A repaired configuration reaches 14.377 of 14.379 bits, above the 13.865-bit oracle; reverting the loss to regression drops it to 0.307 bits, confirming it. Yet the repair licenses nothing about reasoning: the target is reducible, since intra-subgraph edges are a deterministic function of node census. The oracle reaches 96.4% of the ceiling, and our largest effect is the learning-rate schedule, not architecture. Bits and a reasoning probe show no relation across ten cells. A data-derived target fails a quality gate - 25.96% of nodes are duplicate placeholders, and the rest is more generic than supporting evidence. Rank, probes, and metrics can all saturate on an unsupportive evaluation. We release a harness with a reducibility audit and target gate.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 19 of its 24 references that are in the map. First posted on arXiv 20 Aug 2026.

Builds on

19 of 24 references.

References · 19

  1. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,750
  2. 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,173
  3. 2021VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learningcited by 1,057
  4. 2023Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecturecited by 1,039
  5. 2019How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddingscited by 784
  6. 2020Prevalence of Neural Collapse during the terminal phase of deep learning trainingcited by 719
  7. 2020Heterogeneous Graph Transformercited by 582
  8. 2021Understanding Dimensional Collapse in Contrastive Self-supervised Learningcited by 378
  9. 2021Graph Neural Networks with Learnable Structural and Positional Representationscited by 333
  10. 2019Representation Degeneration Problem in Training Natural Language Generation Modelscited by 255
  11. 2021Whitening Sentence Representations for Better Semantics and Faster Retrievalcited by 202
  12. 2022How Powerful are Spectral Graph Neural Networkscited by 169
7 more
  1. 2022RankMe: Assessing the downstream performance of pretrained self-supervised representations by their rankcited by 153
  2. 2023GraphMAE2: A Decoding-Enhanced Masked Self-Supervised Graph Learnercited by 132
  3. 2023Introduction to Latent Variable Energy-Based Models: A Path Towards Autonomous Machine Intelligencecited by 56
  4. 2022Perfectly Balanced: Improving Transfer and Robustness of Supervised Contrastive Learningcited by 40
  5. 2023Graph-level Representation Learning with Joint-Embedding Predictive Architecturescited by 22
  6. 2024Empowering Graph Invariance Learning with Deep Spurious Infomaxcited by 14
  7. 2025Project Alexandria: Towards Freeing Scientific Knowledge from Copyright Burdens via LLMscited by 3

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Align-GRAG: Anchor and Rationale Guided Dual Alignment for Graph Retrieval-Augmented Generation
  2. 2026GraphReAct: Reasoning and Acting for Multi-step Graph Inference
  3. 2026Improving LLM Reasoning with Homophily-aware Structural and Semantic Text-Attributed Graph Compression
  4. 2025Lost in Serialization: Invariance and Generalization of LLM Graph Reasoners
  5. 2025JEPA-Reasoner: Decoupling Latent Reasoning from Token Generation
  6. 2024Investigating Representation Universality: Case Study on Genealogical Representations
  7. 2026Colorful Talks with Graphs: Human-Interpretable Graph Encodings for Large Language Models
  8. 2026Graph Property Inference in Small Language Models: Effects of Representation and Reasoning Strategy