mlsanityData to 14 Oct 2026

SafetyRepro: Configuration-Conditional Rank Instability on Alignment Benchmarks

arXiv 2605.25492 ↗First posted 25 May 2026cs.LGcited by 15

Not on the map. Why?

Abstract (arXiv)

Pairwise model comparisons drawn from foundation-model benchmarks ("A is safer than B") are read as quantitative verdicts but hinge on harness choices benchmark papers under-specify. We close one theory-benchmark loop on this primitive: a finite-envelope proposition tying a measurable pairwise-disagreement rate to whether the strict ordering admits a configuration-pair reversal, paired with a commit-stamped evaluation protocol that operationalises it on widely cited alignment benchmarks. On every benchmark we test, configuration choice alone can flip the pairwise verdict; the proposition isolates this strict-reversal failure mode.

Key facts. Cited by 15 papers in the map (data through 14 Oct 2026). Builds on 22 of its 50 references that are in the map. First posted on arXiv 25 May 2026.

Builds on

22 of 50 references.

References · 22

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  3. 2017A Unified Approach to Interpreting Model Predictionscited by 6,555
  4. 2022Large Language Models are Zero-Shot Reasonerscited by 4,374
  5. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,418
  6. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,975
  7. 2023Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formattingcited by 817
  8. 2020CrowS-Pairs: A Challenge Dataset for Measuring Social Biases in Masked Language Modelscited by 710
  9. 2021BBQ: A Hand-Built Bias Benchmark for Question Answeringcited by 701
  10. 2023XSTest: A Test Suite for Identifying Exaggerated Safety Behaviours in Large Language Modelscited by 505
  11. 2023Large Language Models Are Not Robust Multiple Choice Selectorscited by 486
  12. 2022ToxiGen: A Large-Scale Machine-Generated Dataset for Adversarial and Implicit Hate Speech Detectioncited by 422
10 more
  1. 2021AI and the Everything in the Whole Wide World Benchmarkcited by 349
  2. 2023State of What Art? A Call for Multi-Prompt LLM Evaluationcited by 310
  3. 2019Measurement and Fairnesscited by 308
  4. 2021Accounting for Variance in Machine Learning Benchmarkscited by 201
  5. 2024Lessons from the Trenches on Reproducible Evaluation of Language Modelscited by 179
  6. 2019Show Your Work: Improved Reporting of Experimental Resultscited by 169
  7. 2025AgentAuditor: Human-Level Safety and Security Evaluation for LLM Agentscited by 90
  8. 2024Scaling Law for Time Series Forecastingcited by 33
  9. 2024Mitigating the Risk of Health Inequity Exacerbated by Large Language Modelscited by 21
  10. 2025Attention Consistency for LLMs Explanationcited by 11

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 15.

15 papers.

Sort:

By area:

  1. 2026ChainCaps: Composition-Safe Tool-Using Agents via Monotonic Capability Attenuationcited by 11
  2. 2026When AUC 0.998 Is Not Enough: A Candidate Evaluation Protocol for Hidden-State Probes of Indirect Prompt Injection in Multimodal Computer-Use Agentscited by 7
  3. 2026Phantom Guardrails: When Self-Improving Agent Harnesses Fix Failures That Never Happenedcited by 7
  4. 2026MasDrift: Benchmarking Authorization Preservation Across Multi-Agent Architecturescited by 4
  5. 2026When Should Active RAG Retrieve? A Budget-Aware Evaluation of Utility, Calibration, and Costcited by 2
  6. 2026AutoCRAT: Within-trajectory Joint Control of Stochasticity and Compute for LLM Reasoningcited by 2
  7. 2026Chains That See, Answers That Don't: A Multi-Aspect Evaluation Recipe for Forced Chain-of-Thought on Video-MMEcited by 1
  8. 2026Auditing the Audit: Five Failure Modes in Benchmark-Validity Auditscited by 1
  9. 2026Probe Choice Changes Canary-Memorization Verdicts: Three Post-Hoc Disagreement Case Studies in a Text-Dominant LoRA-Tuned Autoregressive Testbedcited by 0
  10. 2026Where Privacy Risk Lives in English-Source Multilingual RAG: A Stage-Decomposed Audit Across Five Query Languagescited by 0
  11. 2026Mind the Hook: Source-Level Auditing of Privacy Defenses in Retrieval-Augmented Generationcited by 0
  12. 2026Exploratory As-Analyzed No-Detection of Culturally-Marked Predicate-Triggered PII Amplification in a Synthetic-English RAG Probe: A Predicate-Resource-Confounded Auditcited by 0
3 more
  1. 2026Noise Floor Audit for Agent Benchmarkscited by 0
  2. 2026Counterfactual Rollout Replay: Forkable Environments as Free Process Rewards for Software Engineering Agentscited by 0
  3. 2026Revocation Is Not Complete at Receipt: An Executable Outcome Contract for Distributed Agent-Tool Gatewayscited by 0