mlsanityData to 14 Oct 2026

The Growing Pains of Frontier Models: When Leaderboards Stop Separating and What to Measure Next

arXiv 2605.18840 ↗First posted 13 May 2026cs.LG cs.AI cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

Leaderboards rank frontier models on independent axes but do not reveal whether capabilities reinforce or trade off across releases -- and at the frontier, this interaction is the more informative signal. We decompose paired SWE-bench and GPQA Diamond scores into a population coupling trend and per-release residual (h-field) that diagnoses capability emphasis from two public benchmark scores. Across 34 models from 10 labs (2024--2026), capabilities cooperate (r = +0.72, p < 10-6), but cooperation varies systematically: per-lab coupling slopes span 5× (Google 1.15 vs. DeepSeek 0.23), and labs pivot -- DeepSeek reversed from reasoning-rich to coding-first (Delta h = 15.9~pp); Anthropic oscillates between coding excursions and recovery. The population regression serves as an isocline phase boundary: the same √(a/b)· B1 classifier that identifies the base-scale coupling transition [Amin, 2026] classifies frontier models and already detects mixed-phase behavior at the next transition (two models below the GPQA--IFEval isocline). The h-field is not just diagnostic -- it tells you what to change. Pretraining establishes coupling at 0.871 while RLHF adds 0.081 [Amin, 2026]: pretraining-level shifts are permanent (DeepSeek's four-release reversal persists), post-training shifts are reversible (Anthropic's three coding excursions each recover within one release), and inference compute alone shifts h by +7.8~pp without retraining. Knowing which component dominates determines whether to retrain or wait. We provide a three-step diagnostic (locate, classify, predict), a per-lab measurement-priority table, and seven falsifiable predictions with timestamped criteria. Five post-cutoff releases fall within the 95% prediction interval. Code, data, and an interactive dashboard: https://zehenlabs.com/cape/.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 12 of its 22 references that are in the map. First posted on arXiv 13 May 2026.

Builds on

12 of 22 references.

References · 12

  1. 2020Scaling Laws for Neural Language Modelscited by 6,586
  2. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,373
  3. 2022Training Compute-Optimal Large Language Modelscited by 3,019
  4. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,879
  5. 2022Emergent Abilities of Large Language Modelscited by 2,512
  6. 2024Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preferencecited by 1,184
  7. 2023Are Emergent Abilities of Large Language Models a Mirage?cited by 523
  8. 2025Humanity's Last Examcited by 491
  9. 2021Dynabench: Rethinking Benchmarking in NLPcited by 459
  10. 2024Observational Scaling Laws and the Predictability of Language Model Performancecited by 130
  11. 2026When AI Benchmarks Plateau: A Systematic Study of Benchmark Saturationcited by 33
  12. 2023An effective theory of collective deep learningcited by 7

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Coding Agents Have Converged: Why the SWE-bench Leaderboard Can No Longer Order Its Top Entries, and What to Measure Instead
  2. 2026The Capability Frontier: Benchmarks Miss 82% of Model Performance
  3. 2026Bayesian Inference and Decision Audits for Public Archives of Frontier AI Evaluations
  4. 2026Frontier AI Forecasting Has a Measurement Problem: An Audit of Progress Evidence
  5. 2026A Unified Perturbation Framework for Analyzing Leaderboard Stability and Manipulation
  6. 2026Frontier Lag: A Bibliometric Audit of Capability Misrepresentation in Academic AI Evaluation
  7. 2026From Behavior to Mechanism: Tracing Divergent Response Modes in Frontier Language Models
  8. 2026CurveShift: Is Agent Progress Scalar? Separating Level from Shape