# Beyond Words: On Large Language Models Actionability in Mission-Critical Risk Analysis

arXiv: https://arxiv.org/abs/2406.10273 · HTML: https://mlsanity.com/paper/2406.10273/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Placed in Privacy and security of RAG, one of 15 directions in Retrieval-augmented generation.
- Builds on 19 of its 57 references that are in the map.
- First posted on arXiv 11 Jun 2024.

## Placed in

- Retrieval-augmented generation / [Privacy and security of RAG](https://mlsanity.com/rag/privacy-and-security-of-rag/)

## Builds on (most cited first, 19 of 19)

- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [Explainable Artificial Intelligence (XAI): Concepts, Taxonomies, Opportunities and Challenges toward Responsible AI](https://mlsanity.com/paper/1910.10045/) (2019)
- [Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agents](https://mlsanity.com/paper/2201.07207/) (2022)
- [Atlas: Few-shot Learning with Retrieval Augmented Language Models](https://mlsanity.com/paper/2208.03299/) (2022)
- [Large Language Models for Software Engineering: A Systematic Literature Review](https://mlsanity.com/paper/2308.10620/) (2023)
- [R-Judge: Benchmarking Safety Risk Awareness for LLM Agents](https://mlsanity.com/paper/2401.10019/) (2024)
- [OpenAGI: When LLM Meets Domain Experts](https://mlsanity.com/paper/2304.04370/) (2023)
- [Generation-Augmented Retrieval for Open-domain Question Answering](https://mlsanity.com/paper/2009.08553/) (2020)
- [LLM Lies: Hallucinations are not Bugs, but Features as Adversarial Examples](https://mlsanity.com/paper/2310.01469/) (2023)
- [A Review on Language Models as Knowledge Bases](https://mlsanity.com/paper/2204.06031/) (2022)
- [Calibration and Correctness of Language Models for Code](https://mlsanity.com/paper/2402.02047/) (2024)
- [Large Language Models as Tax Attorneys: A Case Study in Legal Capabilities Emergence](https://mlsanity.com/paper/2306.07075/) (2023)
- [CHATREPORT: Democratizing Sustainability Disclosure Analysis through LLM-based Tools](https://mlsanity.com/paper/2307.15770/) (2023)
- [Directive Explanations for Actionable Explainability in Machine Learning Applications](https://mlsanity.com/paper/2102.02671/) (2021)
- [Rocks Coding, Not Development--A Human-Centric, Experimental Evaluation of LLM-Supported SE Tasks](https://mlsanity.com/paper/2402.05650/) (2024)
- [An Extensive Comparison of Static Application Security Testing Tools](https://mlsanity.com/paper/2403.09219/) (2024)
- [Leveraging Large Language Models for Preliminary Security Risk Analysis: A Mission-Critical Case Study](https://mlsanity.com/paper/2403.15756/) (2024)
- [Beware the Rationalization Trap! When Language Model Explainability Diverges from our Mental Models of Language](https://mlsanity.com/paper/2207.06897/) (2022)
- [Ethics in the Age of AI: An Analysis of AI Practitioners' Awareness and Challenges](https://mlsanity.com/paper/2307.10057/) (2023)

## Built on by (most cited first, 9 of 9)

- [Generative AI for Software Architecture. Applications, Challenges, and Future Directions](https://mlsanity.com/paper/2503.13310/) (2025)
- [On Large Language Models in Mission-Critical IT Governance: Are We Ready Yet?](https://mlsanity.com/paper/2412.11698/) (2024)
- [Towards Evaluating Proactive Risk Awareness of Multimodal Language Models](https://mlsanity.com/paper/2505.17455/) (2025)
- [6GSoft: Software for Edge-to-Cloud Continuum](https://mlsanity.com/paper/2407.05963/) (2024)
- [What Were You Thinking? An LLM-Driven Large-Scale Study of Refactoring Motivations in Open-Source Projects](https://mlsanity.com/paper/2509.07763/) (2025)
- [Hacia la interpretabilidad de la detecci'on anticipada de riesgos de depresi'on utilizando grandes modelos de lenguaje](https://mlsanity.com/paper/2503.20939/) (2025)
- [Descriptor: C++ Self-Admitted Technical Debt Dataset (CppSATD)](https://mlsanity.com/paper/2505.01136/) (2025)
- [Evaluating Large Language Models for Detecting Architectural Decision Violations](https://mlsanity.com/paper/2602.07609/) (2026)
- [From Sports to Safety: Benchmarking Proactive Risk Inference in MLLMs](https://mlsanity.com/paper/2608.05560/) (2026)

## Abstract (arXiv)

Context. Risk analysis assesses potential risks in specific scenarios. Risk analysis principles are context-less; the same methodology can be applied to a risk connected to health and information technology security. Risk analysis requires a vast knowledge of national and international regulations and standards and is time and effort-intensive. A large language model can quickly summarize information in less time than a human and can be fine-tuned to specific tasks. Aim. Our empirical study aims to investigate the effectiveness of Retrieval-Augmented Generation and fine-tuned LLM in risk analysis. To our knowledge, no prior study has explored its capabilities in risk analysis. Method. We manually curated 193 unique scenarios leading to 1283 representative samples from over 50 mission-critical analyses archived by the industrial context team in the last five years. We compared the base GPT-3.5 and GPT-4 models versus their Retrieval-Augmented Generation and fine-tuned counterparts. We employ two human experts as competitors of the models and three other human experts to review the models and the former human experts' analysis. The reviewers analyzed 5,000 scenario analyses. Results and Conclusions. Human experts demonstrated higher accuracy, but LLMs are quicker and more actionable. Moreover, our findings show that RAG-assisted LLMs have the lowest hallucination rates, effectively uncovering hidden risks and complementing human expertise. Thus, the choice of model depends on specific needs, with FTMs for accuracy, RAG for hidden risks discovery, and base models for comprehensiveness and actionability. Therefore, experts can leverage LLMs as an effective complementing companion in risk analysis within a condensed timeframe. They can also save costs by averting unnecessary expenses associated with implementing unwarranted countermeasures.