# Agentic and Generative AI for Open-Source Intelligence and Cyber Investigations: Taxonomy, Evaluation, Challenges, and Future Directions

arXiv: https://arxiv.org/abs/2607.03233 · HTML: https://mlsanity.com/paper/2607.03233/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Medical and scientific domain agents, one of 16 directions in LLM agents and tool use.
- Builds on 25 of its 74 references that are in the map.
- First posted on arXiv 3 Jul 2026.

## Placed in

- LLM agents and tool use / [Medical and scientific domain agents](https://mlsanity.com/agents/medical-and-scientific-domain-agents/)

## Builds on (most cited first, 25 of 25)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Training Compute-Optimal Large Language Models](https://mlsanity.com/paper/2203.15556/) (2022)
- [Emergent Abilities of Large Language Models](https://mlsanity.com/paper/2206.07682/) (2022)
- [A Comprehensive Overview of Large Language Models](https://mlsanity.com/paper/2307.06435/) (2023)
- [Verify-and-Edit: A Knowledge-Enhanced Chain-of-Thought Framework](https://mlsanity.com/paper/2305.03268/) (2023)
- [CyberMetric: A Benchmark Dataset based on Retrieval-Augmented Generation for Evaluating LLMs in Cybersecurity Knowledge](https://mlsanity.com/paper/2402.07688/) (2024)
- [Advancements in Generative AI: A Comprehensive Review of GANs, GPT, Autoencoders, Diffusion Model, and Transformers](https://mlsanity.com/paper/2311.10242/) (2023)
- [A Comprehensive Overview of Large Language Models (LLMs) for Cyber Defences: Opportunities and Directions](https://mlsanity.com/paper/2405.14487/) (2024)
- [Recent Advances in Generative AI and Large Language Models: Current Status, Challenges, and Perspectives](https://mlsanity.com/paper/2407.14962/) (2024)
- [Generating Fake Cyber Threat Intelligence Using Transformer-Based Models](https://mlsanity.com/paper/2102.04351/) (2021)
- [At the Dawn of Generative AI Era: A Tutorial-cum-Survey on New Frontiers in 6G Wireless Intelligence](https://mlsanity.com/paper/2402.18587/) (2024)
- [Reasoning Beyond Limits: Advances and Open Problems for LLMs](https://mlsanity.com/paper/2503.22732/) (2025)
- [Evaluation of LLM Chatbots for OSINT-based Cyber Threat Awareness](https://mlsanity.com/paper/2401.15127/) (2024)
- [GeoLocator: a location-integrated large multimodal model for inferring geo-privacy](https://mlsanity.com/paper/2311.13018/) (2023)
- [ThreatModeling-LLM: Automating Threat Modeling using Large Language Models for Banking System](https://mlsanity.com/paper/2411.17058/) (2024)
- [Review of Generative AI Methods in Cybersecurity](https://mlsanity.com/paper/2403.08701/) (2024)
- [GroverGPT: A Large Language Model with 8 Billion Parameters for Quantum Searching](https://mlsanity.com/paper/2501.00135/) (2024)
- [Tracing Thought: Using Chain-of-Thought Reasoning to Identify the LLM Behind AI-Generated Text](https://mlsanity.com/paper/2504.16913/) (2025)
- [OSINT Clinic: Co-designing AI-Augmented Collaborative OSINT Investigations for Vulnerability Assessment](https://mlsanity.com/paper/2409.11672/) (2024)
- [Evaluating the Usability of LLMs in Threat Intelligence Enrichment](https://mlsanity.com/paper/2409.15072/) (2024)
- [Do Generative AI Tools Ensure Green Code? An Investigative Study](https://mlsanity.com/paper/2506.08790/) (2025)
- [CyberThreat-Eval: Can Large Language Models Automate Real-World Threat Research?](https://mlsanity.com/paper/2603.09452/) (2026)
- [OSINT or BULLSHINT? Exploring Open-Source Intelligence tweets about the Russo-Ukrainian War](https://mlsanity.com/paper/2508.03599/) (2025)
- [Forecasting Russian Equipment Losses Using Time Series and Deep Learning Models](https://mlsanity.com/paper/2509.07813/) (2025)

## Abstract (arXiv)

The rapid growth of publicly available digital information has rendered manual open-source intelligence (OSINT) analysis insufficient for modern intelligence, cybersecurity, and cyber investigation. Large language models (LLMs) and agentic AI systems, capable of tool use, multi-step reasoning, and iterative intelligence generation, have emerged as promising solutions, yet evaluation frameworks have not kept pace with reported capabilities. This survey systematically reviews 74 studies and makes four contributions. First, it establishes agentic AI as a distinct analytical category rather than an extension of LLM prompting, organising the literature through an 11-category taxonomy covering LLM foundations, agentic architectures, retrieval-augmented generation (RAG), knowledge graphs, prompt engineering, domain adaptation, evaluation benchmarks, and risk. Second, it identifies the hallucination-validation gap as a corpus-level finding: although hallucination is recognised as a major reliability concern in over twenty studies, end-to-end hallucination is empirically measured in only one OSINT-specific RAG-based system, non-reproducible conditions, while related reasoning and factual-correction studies evaluate general-domain question answering rather than OSINT. Third, it maps existing research to the OSINT lifecycle, showing strong support for collection and analysis but limited coverage of verification, reporting, dissemination, and decision support. Fourth, it derives a ten-point research agenda addressing evaluation, benchmarking, hallucination measurement, adversarial robustness, dark-web coverage, multimodal intelligence, and governance. It concludes that a human-AI co-pilot model, where LLMs assist collection and triage while analysts retain responsibility for verification and decision-making, represents the most defensible near-term deployment architecture.