mlsanityData to 14 Oct 2026

Likelihood scoring for continuations of mathematical text: a self-supervised benchmark with tests for shortcut vulnerabilities

arXiv 2605.10810 ↗First posted 11 May 2026cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

We introduce an automatically generated benchmark for predicting hidden text in technical papers. A paper supplies visible context X and a hidden continuation Y; the evaluated model writes an auxiliary forecast string Z, and a separate scorer assigns next-token probability to Y both with and without conditioning on Z. This gives a label-free test of whether Z transmits information about the continuation, compared against controls where Z is recent context rather than a forecast. Our main testbed is equation-suffix prediction: the predictor sees context and the first part of a displayed equation, then forecasts the rest. The task mixes surface-level arXiv/TeX text modeling with reasoning-sensitive inference; the suffix is one of many roughly equivalent continuations, so the benchmark is read statistically rather than item-by-item. On 1363 equation continuations from 138 recent physics and mathematics papers, forecasts from GPT-5.5, Opus 4.7, and GPT-5.4 nano all improve clipped likelihood over the context control under both Qwen3-8B and Kimi K2.6 scorers, distinguishing model families and reasoning-effort settings without human labels. To emulate shortcuts where Z further primes the scorer rather than making a useful forecast, we also fine-tune the scorer on context-only prompts and apply it to held-out papers as a stronger control. GPT-5.5 forecasts still beat this fine-tuned control; GPT-5.4 nano forecasts do not. Longer prose/TeX continuations show positive but noisier lift over controls, concentrated near the beginning of the target. These results support cross-model likelihood scoring as a static benchmark and as a setup for probing shortcut vulnerabilities before reinforcement learning or model-selection optimization is applied.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 50 of its 64 references that are in the map. First posted on arXiv 11 May 2026.

Builds on

50 of 64 references.

References · 50

  1. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  2. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  3. 2025Qwen3 Technical Reportcited by 8,820
  4. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  5. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,330
  6. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  7. 2023Let's Verify Step by Stepcited by 3,789
  8. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scalecited by 2,810
  9. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,276
  10. 2024Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameterscited by 1,986
  11. 2020Shortcut Learning in Deep Neural Networkscited by 1,969
  12. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,095
38 more
  1. 2024Tulu 3: Pushing Frontiers in Open Language Model Post-Trainingcited by 1,053
  2. 2016The LAMBADA dataset: Word prediction requiring a broad discourse contextcited by 992
  3. 2021BARTScore: Evaluating Generated Text as Text Generationcited by 722
  4. 2024LLM Evaluators Recognize and Favor Their Own Generationscited by 686
  5. 2023Prometheus: Inducing Fine-grained Evaluation Capability in Language Modelscited by 575
  6. 2025Kimi K2: Open Agentic Intelligencecited by 567
  7. 2024Generative Verifiers: Reward Modeling as Next-Token Predictioncited by 480
  8. 2024Quiet-STaR: Language Models Can Teach Themselves to Think Before Speakingcited by 301
  9. 2024LiveBench: A Challenging, Contamination-Limited LLM Benchmarkcited by 262
  10. 2025Logic-RL: Unleashing LLM Reasoning with Rule-Based Reinforcement Learningcited by 262
  11. 2024Scaling Laws for Reward Model Overoptimization in Direct Alignment Algorithmscited by 143
  12. 2023Amortizing intractable inference in large language modelscited by 97
  13. 2023LLMs as Narcissistic Evaluators: When Ego Inflates Evaluation Scorescited by 90
  14. 2024Prover-Verifier Games improve legibility of LLM outputscited by 85
  15. 2025RLPR: Extrapolating RLVR to General Domains without Verifierscited by 85
  16. 2025Reinforcing General Reasoning without Verifierscited by 77
  17. 2025One Token to Fool LLM-as-a-Judgecited by 60
  18. 2023Training Chain-of-Thought via Latent-Variable Inferencecited by 47
  19. 2024AntiLeakBench: Preventing Data Contamination by Automatically Constructing Benchmarks with Updated Real-World Knowledgecited by 44
  20. 2025Reinforcement Pre-Trainingcited by 40
  21. 2025Learning to Reason for Long-Form Story Generationcited by 34
  22. 2025Inference-Time Reward Hacking in Large Language Modelscited by 34
  23. 2025Beyond Verifiable Rewards: Scaling Reinforcement Learning for Language Models to Unverifiable Datacited by 29
  24. 2026LLMs Gaming Verifiers: RLVR can Lead to Reward Hackingcited by 28
  25. 2025RLP: Reinforcement as a Pretraining Objectivecited by 25
  26. 2024Benchmarking LLMs' Judgments with No Gold Standardcited by 19
  27. 2025Variation in Verification: Understanding Verification Dynamics in Large Language Modelscited by 19
  28. 2025Direct Reasoning Optimization: Token-Level Reasoning Reflectivity Meets Rubric Gates for Unverifiable Taskscited by 18
  29. 2024Eliciting Informative Text Evaluations with Large Language Modelscited by 15
  30. 2025NOVER: Incentive Training for Language Models via Verifier-Free Reinforcement Learningcited by 15
  31. 2025Reinforcement Learning on Pre-Training Datacited by 12
  32. 2018Eliciting Expertise without Verificationcited by 11
  33. 2026Learning to Simulate Human Dialoguecited by 9
  34. 2026Reward Under Attack: Analyzing the Robustness and Hackability of Process Reward Modelscited by 9
  35. 2026Likelihood-Based Reward Designs for General LLM Reasoningcited by 7
  36. 2025How Much Do Large Language Model Cheat on Evaluation? Benchmarking Overestimation under the One-Time-Pad-Based Frameworkcited by 6
  37. 2025BOW: Training Language Models to Reason Over Plausible Next Wordscited by 2
  38. 2025From Raw Corpora to Domain Benchmarks: Automated Evaluation of LLM Domain Expertisecited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026DiffScore: Text Evaluation Beyond Autoregressive Likelihood
  2. 2026Forking Fast: Efficiently Estimating Uncertainty Dynamics in Text Generation
  3. 2026Towards Computational Provenance: Carrying Causal-State Evidence in Generated Text
  4. 2025Can an LLM Induce a Graph? Investigating Memory Drift and Context Length
  5. 2026Robust Reasoning Benchmark
  6. 2025Multiple Token Divergence: Measuring and Steering In-Context Computation Density
  7. 2023OpenWebMath: An Open Dataset of High-Quality Mathematical Web Text
  8. 2024NeedleBench: Evaluating LLM Retrieval and Reasoning Across Varying Information Densities