mlsanityData to 14 Oct 2026

Silent Revision: Measuring Undisclosed Change in the Safety Frameworks of Frontier AI Developers

arXiv 2609.08789 ↗First posted 8 Sep 2026cs.CY cs.AI cs.SEcited by 0

Not on the map. Why?

Abstract (arXiv)

Frontier AI developers publish safety frameworks that commit them to evidencing whether their models are dangerous. The European Union and California now treat these documents as instruments of accountability, and both already impose duties on their revision. Neither requires the revision to be legible, in the sense that a reader could learn from the developer's own account what changed. We introduce the silent revision rate, the share of material changes to a framework's commitments that the developer's published account does not identify, and we release the versioned, hash-pinned corpus needed to compute it. The corpus contains every public version of the safety frameworks of the twelve developers that have published one, together with each provider's changelog, redline or announcement. We trace 710 commitment instances across twelve consecutive version pairs, code them against a frozen codebook, and adjudicate 244 individually. Three findings follow. First, 67% of material changes (95% CI 62 to 72) are silent under a strict standard and 53% under a lenient one, falling to 49% at section granularity. Second, silence appears to track the form of the account, since narrative announcements run at 74% against 63% for itemised changelogs, whereas account length in words barely matters; on the test that respects nesting the difference is suggestive. Third, 77% of traced changes weaken or remove a commitment, and in seven of eight pairs weakenings are more often silent than strengthenings. The statutory remedy therefore exists and specifies the wrong artefact. A justification explains why a framework changed, an enumeration states what changed, and only the latter makes revision auditable. We argue that publication duties should carry an enumeration duty, which one provider already meets, voluntarily and incompletely.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 27 of its 52 references that are in the map. First posted on arXiv 8 Sep 2026.

Builds on

27 of 52 references.

References · 27

  1. 2018Datasheets for Datasetscited by 1,386
  2. 2018Model Cards for Model Reportingcited by 1,080
  3. 2023ChatGPT Outperforms Crowd-Workers for Text-Annotation Taskscited by 870
  4. 2020Closing the AI Accountability Gap: Defining an End-to-End Framework for Internal Algorithmic Auditingcited by 384
  5. 2019Measurement and Fairnesscited by 308
  6. 2024Political Compass or Spinning Arrow? Towards More Meaningful Evaluations for Values and Opinions in Large Language Modelscited by 162
  7. 2019Principles alone cannot guarantee ethical AIcited by 135
  8. 2023Frontier AI Regulation: Managing Emerging Risks to Public Safetycited by 98
  9. 2024SafetyPrompts: a Systematic Review of Open Datasets for Evaluating and Improving Large Language Model Safetycited by 77
  10. 2023Automated Annotation with Generative AI Requires Validationcited by 76
  11. 2022Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governancecited by 67
  12. 2023The Foundation Model Transparency Indexcited by 66
15 more
  1. 2025Position: Evaluating Generative AI Systems Is a Social Science Measurement Challengecited by 63
  2. 2023Who Audits the Auditors? Recommendations from a field scan of the algorithmic auditing ecosystemcited by 52
  3. 2025Toward an Evaluation Science for Generative AI Systemscited by 45
  4. 2020Privacy Policies over Time: Curation and Analysis of a Million-Document Datasetcited by 28
  5. 2023Towards best practices in AGI safety and governance: A survey of expert opinioncited by 27
  6. 2024The 2024 Foundation Model Transparency Indexcited by 19
  7. 2024Foundation Model Transparency Reportscited by 14
  8. 2025The 2025 Foundation Model Transparency Indexcited by 11
  9. 2024Responsible Reporting for Frontier AI Developmentcited by 10
  10. 2024Risk thresholds for frontier AIcited by 9
  11. 2025A Frontier AI Risk Management Framework: Bridging the Gap Between Current AI Practices and Established Risk Managementcited by 8
  12. 2024A Grading Rubric for AI Safety Frameworkscited by 5
  13. 2025Evaluating AI Providers' Frontier Safety Frameworkscited by 3
  14. 2024Measurement challenges in AI catastrophic risk governance and safety frameworkscited by 2
  15. 2025Towards Frontier Safety Policies Pluscited by 1

Built on by

Not cited yet.