# System Cards for AI-Based Decision-Making for Public Policy

arXiv: https://arxiv.org/abs/2203.04754 · HTML: https://mlsanity.com/paper/2203.04754/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 130 references that are in the map.
- First posted on arXiv 1 Mar 2022.

## Builds on (most cited first, 25 of 25)

- [A Unified Approach to Interpreting Model Predictions](https://mlsanity.com/paper/1705.07874/) (2017)
- ["Why Should I Trust You?": Explaining the Predictions of Any Classifier](https://mlsanity.com/paper/1602.04938/) (2016)
- [A Survey on Bias and Fairness in Machine Learning](https://mlsanity.com/paper/1908.09635/) (2019)
- [Federated Learning: Challenges, Methods, and Future Directions](https://mlsanity.com/paper/1908.07873/) (2019)
- [Datasheets for Datasets](https://mlsanity.com/paper/1803.09010/) (2018)
- [Model Cards for Model Reporting](https://mlsanity.com/paper/1810.03993/) (2018)
- [Inherent Trade-Offs in the Fair Determination of Risk Scores](https://mlsanity.com/paper/1609.05807/) (2016)
- [Explaining Explanations: An Overview of Interpretability of Machine Learning](https://mlsanity.com/paper/1806.00069/) (2018)
- [Closing the AI Accountability Gap: Defining an End-to-End Framework for Internal Algorithmic Auditing](https://mlsanity.com/paper/2001.00973/) (2020)
- [Fairness in Criminal Justice Risk Assessments: The State of the Art](https://mlsanity.com/paper/1703.09207/) (2017)
- [Aequitas: A Bias and Fairness Audit Toolkit](https://mlsanity.com/paper/1811.05577/) (2018)
- [The Hidden Assumptions Behind Counterfactual Explanations and Principal Reasons](https://mlsanity.com/paper/1912.04930/) (2019)
- [Why Fairness Cannot Be Automated: Bridging the Gap Between EU Non-Discrimination Law and AI](https://mlsanity.com/paper/2005.05906/) (2020)
- [Interpretability and Explainability: A Machine Learning Zoo Mini-tour](https://mlsanity.com/paper/2012.01805/) (2020)
- [Everyday algorithm auditing: Understanding the power of everyday users in surfacing harmful algorithmic behaviors](https://mlsanity.com/paper/2105.02980/) (2021)
- [Data Poisoning Attacks against Online Learning](https://mlsanity.com/paper/1808.08994/) (2018)
- [Outlining Traceability: A Principle for Operationalizing Accountability in Computing Systems](https://mlsanity.com/paper/2101.09385/) (2021)
- [Problematic Machine Behavior: A Systematic Literature Review of Algorithm Audits](https://mlsanity.com/paper/2102.04256/) (2021)
- [On the Legal Compatibility of Fairness Definitions](https://mlsanity.com/paper/1912.00761/) (2019)
- [Responsible Autonomy](https://mlsanity.com/paper/1706.02513/) (2017)
- [A Survey on AI Assurance](https://mlsanity.com/paper/2111.07505/) (2021)
- [Enslaving the Algorithm: From a "Right to an Explanation" to a "Right to Better Decisions"?](https://mlsanity.com/paper/1803.07540/) (2018)
- [Ethics-Based Auditing of Automated Decision-Making Systems: Nature, Scope, and Limitations](https://mlsanity.com/paper/2110.10980/) (2021)
- [Towards Accountability in the Use of Artificial Intelligence for Public Administrations](https://mlsanity.com/paper/2105.01434/) (2021)
- [Next Wave Artificial Intelligence: Robust, Explainable, Adaptable, Ethical, and Accountable](https://mlsanity.com/paper/2012.06058/) (2020)

## Built on by (most cited first, 13 of 13)

- [The 2025 AI Agent Index: Documenting Technical and Safety Features of Deployed Agentic AI Systems](https://mlsanity.com/paper/2602.17753/) (2026)
- [Market Concentration Implications of Foundation Models](https://mlsanity.com/paper/2311.01550/) (2023)
- [STREAM (ChemBio): A Standard for Transparently Reporting Evaluations in AI Model Reports](https://mlsanity.com/paper/2508.09853/) (2025)
- [Eval Factsheets: A Structured Framework for Documenting AI Evaluations](https://mlsanity.com/paper/2512.04062/) (2025)
- [Error Parity Fairness: Testing for Group Fairness in Regression Tasks](https://mlsanity.com/paper/2208.08279/) (2022)
- [Evaluation Cards: An Interpretive Layer for AI Evaluation Reporting](https://mlsanity.com/paper/2606.09809/) (2026)
- [Equal Confusion Fairness: Measuring Group-Based Disparities in Automated Decision Systems](https://mlsanity.com/paper/2307.00472/) (2023)
- [Both eyes open: Vigilant Incentives help Regulatory Markets improve AI Safety](https://mlsanity.com/paper/2303.03174/) (2023)
- [Mitigating Downstream Model Risks via Model Provenance](https://mlsanity.com/paper/2410.02230/) (2024)
- [AI Data Development: A Scorecard for the System Card Framework](https://mlsanity.com/paper/2506.02071/) (2025)
- [Applications of Risk Science to AI Fairness Evaluation: Principles, Challenges, and Best Practices](https://mlsanity.com/paper/2608.29478/) (2026)
- [The "Collections as ML Data" Checklist for Machine Learning & Cultural Heritage](https://mlsanity.com/paper/2207.02960/) (2022)
- [A Chain Is Only as Strong as Its Weakest Link: A Scoping Review of System Integration Audits in AI](https://mlsanity.com/paper/2608.04921/) (2026)

## Abstract (arXiv)

Decisions impacting human lives are increasingly being made or assisted by automated decision-making algorithms. Many of these algorithms process personal data for predicting recidivism, credit risk analysis, identifying individuals using face recognition, and more. While potentially improving efficiency and effectiveness, such algorithms are not inherently free from bias, opaqueness, lack of explainability, maleficence, and the like. Given that the outcomes of these algorithms have a significant impact on individuals and society and are open to analysis and contestation after deployment, such issues must be accounted for before deployment. Formal audits are a way of ensuring algorithms meet the appropriate accountability standards. This work, based on an extensive analysis of the literature and an expert focus group study, proposes a unifying framework for a system accountability benchmark for formal audits of artificial intelligence-based decision-aiding systems. This work also proposes system cards to serve as scorecards presenting the outcomes of such audits. It consists of 56 criteria organized within a four-by-four matrix composed of rows focused on (i) data, (ii) model, (iii) code, (iv) system, and columns focused on (a) development, (b) assessment, (c) mitigation, and (d) assurance. The proposed system accountability benchmark reflects the state-of-the-art developments for accountable systems, serves as a checklist for algorithm audits, and paves the way for sequential work in future research.