# Designing Safe and Accountable GenAI as a Learning Companion with Women Banned from Formal Education

arXiv: https://arxiv.org/abs/2604.07253 · HTML: https://mlsanity.com/paper/2604.07253/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 18 of its 84 references that are in the map.
- First posted on arXiv 8 Apr 2026.

## Builds on (most cited first, 18 of 18)

- [Closing the AI Accountability Gap: Defining an End-to-End Framework for Internal Algorithmic Auditing](https://mlsanity.com/paper/2001.00973/) (2020)
- [Large Language Models for Education: A Survey and Outlook](https://mlsanity.com/paper/2403.18105/) (2024)
- [Automatic Generation of Programming Exercises and Code Explanations using Large Language Models](https://mlsanity.com/paper/2206.11861/) (2022)
- [Black-Box Access is Insufficient for Rigorous AI Audits](https://mlsanity.com/paper/2401.14446/) (2024)
- [Practical and Ethical Challenges of Large Language Models in Education: A Systematic Scoping Review](https://mlsanity.com/paper/2303.13379/) (2023)
- [Accountability in an Algorithmic Society: Relationality, Responsibility, and Robustness in Machine Learning](https://mlsanity.com/paper/2202.05338/) (2022)
- [Engaging with Massive Online Courses](https://mlsanity.com/paper/1403.3100/) (2014)
- ["Don't Forget the Teachers": Towards an Educator-Centered Understanding of Harms from Large Language Models in Education](https://mlsanity.com/paper/2502.14592/) (2025)
- [GPTutor: Great Personalized Tutor with Large Language Models for Personalized Learning Content Generation](https://mlsanity.com/paper/2407.09484/) (2024)
- [The Effects of GitHub Copilot on Computing Students' Programming Effectiveness, Efficiency, and Processes in Brownfield Programming Tasks](https://mlsanity.com/paper/2506.10051/) (2025)
- [Comparative Explanations of Recommendations](https://mlsanity.com/paper/2111.00670/) (2021)
- [Charting the Future of AI in Project-Based Learning: A Co-Design Exploration with Students](https://mlsanity.com/paper/2401.14915/) (2024)
- [Low-resourced Languages and Online Knowledge Repositories: A Need-Finding Study](https://mlsanity.com/paper/2405.16669/) (2024)
- [Responsible Adoption of Generative AI in Higher Education: Developing a "Points to Consider" Approach Based on Faculty Perspectives](https://mlsanity.com/paper/2406.01930/) (2024)
- [Co-designing Large Language Model Tools for Project-Based Learning with K12 Educators](https://mlsanity.com/paper/2502.09799/) (2025)
- [Shaping Integrity: Why Generative Artificial Intelligence Does Not Have to Undermine Education](https://mlsanity.com/paper/2407.19088/) (2024)
- ["How can we learn and use AI at the same time?": Participatory Design of GenAI with High School Students](https://mlsanity.com/paper/2506.15525/) (2025)
- [Hope, Aspirations, and the Impact of LLMs on Female Programming Learners in Afghanistan](https://mlsanity.com/paper/2511.08630/) (2025)

## Abstract (arXiv)

In gender-restrictive and surveilled contexts, where access to formal education may be restricted for women, pursuing education involves safety and privacy risks. When women are excluded from schools and universities, they often turn to online self-learning and generative AI (GenAI) to pursue their educational and career aspirations. However, we know little about what safe and accountable GenAI support is required in the context of surveillance, household responsibilities, and the absence of learning communities. We present a remote participatory design study with 20 women in Afghanistan, informed by a recruitment survey (n = 140), examining how participants envision GenAI for learning and employability. Participants describe using GenAI less as an information source and more as an always-available peer, mentor, and source of career guidance that helps compensate for the absence of learning communities. At the same time, they emphasize that this companionship is constrained by privacy and surveillance risks, contextually unrealistic and culturally unsafe support, and direct-answer interactions that can undermine learning by creating an illusion of progress. Beyond eliciting requirements, envisioning the future with GenAI through participatory design was positively associated with significant increases in participants' aspirations (p=.01), perceived agency (p=.01), and perceived avenues (p=.03). These outcomes show that accountable and safe GenAI is not only about harm reduction but can also actively enable women to imagine and pursue viable learning and employment futures. Building on this, we translate participants' proposals into accountability-focused design directions that center on safety-first interaction and user control, context-grounded support under constrained resources, and offer pedagogically aligned assistance that supports genuine learning rather than quick answers.