# Difficult Lessons on Social Prediction from Wisconsin Public Schools

arXiv: https://arxiv.org/abs/2304.06205 · HTML: https://mlsanity.com/paper/2304.06205/

## Key facts

- Cited by 35 papers in the map (data through 14 Oct 2026).
- Builds on 3 of its 32 references that are in the map.
- First posted on arXiv 13 Apr 2023.

## Builds on (most cited first, 3 of 3)

- [Performative Prediction](https://mlsanity.com/paper/2002.06673/) (2020)
- [Anticipating Performativity by Predicting from Predictions](https://mlsanity.com/paper/2208.07331/) (2022)
- [Is your model predicting the past?](https://mlsanity.com/paper/2206.11673/) (2022)

## Built on by (most cited first, 30 of 35)

- [Performative Prediction: Past and Future](https://mlsanity.com/paper/2310.16608/) (2023)
- [Evaluating language models as risk scores](https://mlsanity.com/paper/2407.14614/) (2024)
- [The Relative Value of Prediction in Algorithmic Decision Making](https://mlsanity.com/paper/2312.08511/) (2023)
- [The Benchmarking Epistemology: Validity Theory for Evaluating Machine Learning Models](https://mlsanity.com/paper/2510.23191/) (2025)
- [On the Impossibility of Separating Intelligence from Judgment: The Computational Intractability of Filtering for AI Alignment](https://mlsanity.com/paper/2507.07341/) (2025)
- [Revisiting the Predictability of Performative, Social Events](https://mlsanity.com/paper/2503.11713/) (2025)
- [On the Actionability of Outcome Prediction](https://mlsanity.com/paper/2309.04470/) (2023)
- [Allocation Requires Prediction Only if Inequality Is Low](https://mlsanity.com/paper/2406.13882/) (2024)
- [Performative Prediction on Games and Mechanism Design](https://mlsanity.com/paper/2408.05146/) (2024)
- [Evaluating the Effectiveness of Index-Based Treatment Allocation](https://mlsanity.com/paper/2402.11771/) (2024)
- [The Value of Prediction in Identifying the Worst-Off](https://mlsanity.com/paper/2501.19334/) (2025)
- [Three Types of Calibration with Properties and their Semantic and Formal Relationships](https://mlsanity.com/paper/2504.18395/) (2025)
- [The Costs of Pretending That There Are Data-Generating Probability Distributions in the Social World](https://mlsanity.com/paper/2407.17395/) (2024)
- [Estimating Wage Disparities Using Foundation Models](https://mlsanity.com/paper/2409.09894/) (2024)
- [Comparing Targeting Strategies for Maximizing Social Welfare with Limited Resources](https://mlsanity.com/paper/2411.07414/) (2024)
- [The Stability of Online Algorithms in Performative Prediction](https://mlsanity.com/paper/2602.24207/) (2026)
- [Beyond Eviction Prediction: Leveraging Local Spatiotemporal Public Records to Inform Action](https://mlsanity.com/paper/2401.16440/) (2024)
- [Who Pays the RENT? Implications of Spatial Inequality for Prediction-Based Allocation Policies](https://mlsanity.com/paper/2508.08573/) (2025)
- [Fostering the Ecosystem of AI for Social Impact Requires Expanding and Strengthening Evaluation Standards](https://mlsanity.com/paper/2510.18238/) (2025)
- [Good Allocations from Bad Estimates](https://mlsanity.com/paper/2601.05597/) (2026)
- [Dissecting Performative Prediction: A Comprehensive Survey](https://mlsanity.com/paper/2602.10176/) (2026)
- [The Hidden Cost of Waiting for Accurate Predictions](https://mlsanity.com/paper/2503.00650/) (2025)
- [Reconsidering Fairness Through Unawareness From the Perspective of Model Multiplicity](https://mlsanity.com/paper/2505.16638/) (2025)
- [The Role of Causality in Algorithmic Recourse](https://mlsanity.com/paper/2607.28497/) (2026)
- [Bridging the gap: Towards an Expanded Toolkit for AI-driven Decision-Making in the Public Sector](https://mlsanity.com/paper/2310.19091/) (2023)
- [On Fixing the Right Problems in Predictive Analytics: AUC Is Not the Problem](https://mlsanity.com/paper/2404.06989/) (2024)
- [Position: Beyond Sensitive Attributes, ML Fairness Should Quantify Structural Injustice via Social Determinants](https://mlsanity.com/paper/2508.08337/) (2025)
- [Performative Learning Theory](https://mlsanity.com/paper/2602.04402/) (2026)
- [On the Meta-Design of Allocation Problems](https://mlsanity.com/paper/2602.08786/) (2026)
- [Representation Fidelity:Auditing Algorithmic Decisions About Humans Using Self-Descriptions](https://mlsanity.com/paper/2603.05136/) (2026)

## Abstract (arXiv)

Early warning systems (EWS) are predictive tools at the center of recent efforts to improve graduation rates in public schools across the United States. These systems assist in targeting interventions to individual students by predicting which students are at risk of dropping out. Despite significant investments in their widespread adoption, there remain large gaps in our understanding of the efficacy of EWS, and the role of statistical risk scores in education. In this work, we draw on nearly a decade's worth of data from a system used throughout Wisconsin to provide the first large-scale evaluation of the long-term impact of EWS on graduation outcomes. We present empirical evidence that the prediction system accurately sorts students by their dropout risk. We also find that it may have caused a single-digit percentage increase in graduation rates, though our empirical analyses cannot reliably rule out that there has been no positive treatment effect. Going beyond a retrospective evaluation of DEWS, we draw attention to a central question at the heart of the use of EWS: Are individual risk scores necessary for effectively targeting interventions? We propose a simple mechanism that only uses information about students' environments -- such as their schools, and districts -- and argue that this mechanism can target interventions just as efficiently as the individual risk score-based mechanism. Our argument holds even if individual predictions are highly accurate and effective interventions exist. In addition to motivating this simple targeting mechanism, our work provides a novel empirical backbone for the robust qualitative understanding among education researchers that dropout is structurally determined. Combined, our insights call into question the marginal value of individual predictions in settings where outcomes are driven by high levels of inequality.