# Risks and NLP Design: A Case Study on Procedural Document QA

arXiv: https://arxiv.org/abs/2408.11860 · HTML: https://mlsanity.com/paper/2408.11860/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 37 references that are in the map.
- First posted on arXiv 16 Aug 2024.

## Builds on (most cited first, 12 of 12)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Language (Technology) is Power: A Critical Survey of "Bias" in NLP](https://mlsanity.com/paper/2005.14050/) (2020)
- [Measurement and Fairness](https://mlsanity.com/paper/1912.05511/) (2019)
- [All That's 'Human' Is Not Gold: Evaluating Human Evaluation of Generated Text](https://mlsanity.com/paper/2107.00061/) (2021)
- [Predictability and Surprise in Large Generative Models](https://mlsanity.com/paper/2202.07785/) (2022)
- [FaVIQ: FAct Verification from Information-seeking Questions](https://mlsanity.com/paper/2107.02153/) (2021)
- [Reliability Testing for Natural Language Processing Systems](https://mlsanity.com/paper/2105.02590/) (2021)
- [Multi-Passage Machine Reading Comprehension with Cross-Passage Answer Verification](https://mlsanity.com/paper/1805.02220/) (2018)
- [Calibration of Machine Reading Systems at Scale](https://mlsanity.com/paper/2203.10623/) (2022)
- [Controlling the Risk of Conversational Search via Reinforcement Learning](https://mlsanity.com/paper/2101.06327/) (2021)
- [The Risks of Machine Learning Systems](https://mlsanity.com/paper/2204.09852/) (2022)
- [Controlling Risk of Web Question Answering](https://mlsanity.com/paper/1905.10077/) (2019)

## Built on by (most cited first, 2 of 2)

- [How Performance Pressure Influences AI-Assisted Decision Making](https://mlsanity.com/paper/2410.16560/) (2024)
- [Customized Generative AI Agent for Transportation Engineering Practice: A Development and Continued Pre-training Guideline](https://mlsanity.com/paper/2606.29014/) (2026)

## Abstract (arXiv)

As NLP systems are increasingly deployed at scale, concerns about their potential negative impacts have attracted the attention of the research community, yet discussions of risk have mostly been at an abstract level and focused on generic AI or NLP applications. We argue that clearer assessments of risks and harms to users--and concrete strategies to mitigate them--will be possible when we specialize the analysis to more concrete applications and their plausible users. As an illustration, this paper is grounded in cooking recipe procedural document question answering (ProcDocQA), where there are well-defined risks to users such as injuries or allergic reactions. Our case study shows that an existing language model, applied in "zero-shot" mode, quantitatively answers real-world questions about recipes as well or better than the humans who have answered the questions on the web. Using a novel questionnaire informed by theoretical work on AI risk, we conduct a risk-oriented error analysis that could then inform the design of a future system to be deployed with lower risk of harm and better performance.