# Reproducibility as a Mechanism for Teaching Fairness, Accountability, Confidentiality, and Transparency in Artificial Intelligence

arXiv: https://arxiv.org/abs/2111.00826 · HTML: https://mlsanity.com/paper/2111.00826/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 8 of its 32 references that are in the map.
- First posted on arXiv 1 Nov 2021.

## Builds on (most cited first, 8 of 8)

- ["Why Should I Trust You?": Explaining the Predictions of Any Classifier](https://mlsanity.com/paper/1602.04938/) (2016)
- [Deep Learning with Differential Privacy](https://mlsanity.com/paper/1607.00133/) (2016)
- [Equality of Opportunity in Supervised Learning](https://mlsanity.com/paper/1610.02413/) (2016)
- [Troubling Trends in Machine Learning Scholarship](https://mlsanity.com/paper/1807.03341/) (2018)
- [Order in the Court: Explainable AI Methods Prone to Disagreement](https://mlsanity.com/paper/2105.03287/) (2021)
- [Value Cards: An Educational Toolkit for Teaching Social Impacts of Machine Learning through Deliberation](https://mlsanity.com/paper/2010.11411/) (2020)
- [Teaching Responsible Data Science: Charting New Pedagogical Territory](https://mlsanity.com/paper/1912.10564/) (2019)
- [ReproducedPapers.org: Openly teaching and structuring machine learning reproducibility](https://mlsanity.com/paper/2012.01172/) (2020)

## Built on by (most cited first, 4 of 4)

- [The AI Incident Database as an Educational Tool to Raise Awareness of AI Harms: A Classroom Exploration of Efficacy, Limitations, & Future Improvements](https://mlsanity.com/paper/2310.06269/) (2023)
- [Users are the North Star for AI Transparency](https://mlsanity.com/paper/2303.05500/) (2023)
- [Maestro: A Gamified Platform for Teaching AI Robustness](https://mlsanity.com/paper/2306.08238/) (2023)
- [AI Research is not Magic, it has to be Reproducible and Responsible: Challenges in the AI field from the Perspective of its PhD Students](https://mlsanity.com/paper/2408.06847/) (2024)

## Abstract (arXiv)

In this work, we explain the setup for a technical, graduate-level course on Fairness, Accountability, Confidentiality, and Transparency in Artificial Intelligence (FACT-AI) at the University of Amsterdam, which teaches FACT-AI concepts through the lens of reproducibility. The focal point of the course is a group project based on reproducing existing FACT-AI algorithms from top AI conferences and writing a corresponding report. In the first iteration of the course, we created an open source repository with the code implementations from the group projects. In the second iteration, we encouraged students to submit their group projects to the Machine Learning Reproducibility Challenge, resulting in 9 reports from our course being accepted for publication in the ReScience journal. We reflect on our experience teaching the course over two years, where one year coincided with a global pandemic, and propose guidelines for teaching FACT-AI through reproducibility in graduate-level AI study programs. We hope this can be a useful resource for instructors who want to set up similar courses in the future.