# Improving Ethical Outcomes with Machine-in-the-Loop: Broadening Human Understanding of Data Annotations

arXiv: https://arxiv.org/abs/2112.09738 · HTML: https://mlsanity.com/paper/2112.09738/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 5 of its 11 references that are in the map.
- First posted on arXiv 16 Dec 2021.

## Builds on (most cited first, 5 of 5)

- [A Survey on Bias and Fairness in Machine Learning](https://mlsanity.com/paper/1908.09635/) (2019)
- [A Step Toward More Inclusive People Annotations for Fairness](https://mlsanity.com/paper/2105.02317/) (2021)
- [Algorithmic Fairness from a Non-ideal Perspective](https://mlsanity.com/paper/2001.09773/) (2020)
- [Towards the Right Kind of Fairness in AI](https://mlsanity.com/paper/2102.08453/) (2021)
- [Ideal theory in AI ethics](https://mlsanity.com/paper/2011.02279/) (2020)

## Abstract (arXiv)

We introduce a machine-in-the-loop pipeline that aims to address root causes of unwanted bias in natural language based supervised machine learning tasks in the education domain. Learning from the experiences of students is foundational for education researchers, and academic administrators. 21st-century skills learned from experience are becoming a core part of college and career readiness as well as the hiring process in the new knowledge economy. Minoritized students demonstrate these skills in their daily lives, but documenting, assessing, and validating these skills is a huge problem for educational institutions. As an equity focused online platform, LivedX translates minoritized students' lived experiences into the 21st century skills, issues micro-credentials, and creates personal 21st century skills portfolio. To automate the micro credential mining from the natural language texts received from the students' submitted essays, we employed a bag-of-word model to construct a multi-output classifier. Despite our goal, our model initially exacerbated disparate impact on minoritized students. We used a machine-in-the-loop model development pipeline to address the problem and refine the aforementioned model to ensure fairness in its prediction.