Agentic and Generative AI for Open-Source Intelligence and Cyber Investigations: Taxonomy, Evaluation, Challenges, and Future Directions
On the map: LLM agents and tool use → Medical and scientific domain agents
Abstract (arXiv)
The rapid growth of publicly available digital information has rendered manual open-source intelligence (OSINT) analysis insufficient for modern intelligence, cybersecurity, and cyber investigation. Large language models (LLMs) and agentic AI systems, capable of tool use, multi-step reasoning, and iterative intelligence generation, have emerged as promising solutions, yet evaluation frameworks have not kept pace with reported capabilities. This survey systematically reviews 74 studies and makes four contributions. First, it establishes agentic AI as a distinct analytical category rather than an extension of LLM prompting, organising the literature through an 11-category taxonomy covering LLM foundations, agentic architectures, retrieval-augmented generation (RAG), knowledge graphs, prompt engineering, domain adaptation, evaluation benchmarks, and risk. Second, it identifies the hallucination-validation gap as a corpus-level finding: although hallucination is recognised as a major reliability concern in over twenty studies, end-to-end hallucination is empirically measured in only one OSINT-specific RAG-based system, non-reproducible conditions, while related reasoning and factual-correction studies evaluate general-domain question answering rather than OSINT. Third, it maps existing research to the OSINT lifecycle, showing strong support for collection and analysis but limited coverage of verification, reporting, dissemination, and decision support. Fourth, it derives a ten-point research agenda addressing evaluation, benchmarking, hallucination measurement, adversarial robustness, dark-web coverage, multimodal intelligence, and governance. It concludes that a human-AI co-pilot model, where LLMs assist collection and triage while analysts retain responsibility for verification and decision-making, represents the most defensible near-term deployment architecture.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Medical and scientific domain agents, one of 16 directions in LLM agents and tool use. Builds on 25 of its 74 references that are in the map. First posted on arXiv 3 Jul 2026.
Builds on
25 of 74 references.
References · 25
- 2020Language Models are Few-Shot Learnerscited by 32,199
- 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
- 2022Training Compute-Optimal Large Language Modelscited by 2,927
- 2022Emergent Abilities of Large Language Modelscited by 2,483
- 2023A Comprehensive Overview of Large Language Modelscited by 639
- 2023Verify-and-Edit: A Knowledge-Enhanced Chain-of-Thought Frameworkcited by 139
- 2024CyberMetric: A Benchmark Dataset based on Retrieval-Augmented Generation for Evaluating LLMs in Cybersecurity Knowledgecited by 55
- 2023Advancements in Generative AI: A Comprehensive Review of GANs, GPT, Autoencoders, Diffusion Model, and Transformerscited by 35
- 2024A Comprehensive Overview of Large Language Models (LLMs) for Cyber Defences: Opportunities and Directionscited by 33
- 2024Recent Advances in Generative AI and Large Language Models: Current Status, Challenges, and Perspectivescited by 30
- 2021Generating Fake Cyber Threat Intelligence Using Transformer-Based Modelscited by 28
- 2024At the Dawn of Generative AI Era: A Tutorial-cum-Survey on New Frontiers in 6G Wireless Intelligencecited by 25
13 more
- 2025Reasoning Beyond Limits: Advances and Open Problems for LLMscited by 16
- 2024Evaluation of LLM Chatbots for OSINT-based Cyber Threat Awarenesscited by 14
- 2023GeoLocator: a location-integrated large multimodal model for inferring geo-privacycited by 13
- 2024ThreatModeling-LLM: Automating Threat Modeling using Large Language Models for Banking Systemcited by 10
- 2024Review of Generative AI Methods in Cybersecuritycited by 9
- 2024GroverGPT: A Large Language Model with 8 Billion Parameters for Quantum Searchingcited by 7
- 2025Tracing Thought: Using Chain-of-Thought Reasoning to Identify the LLM Behind AI-Generated Textcited by 3
- 2024OSINT Clinic: Co-designing AI-Augmented Collaborative OSINT Investigations for Vulnerability Assessmentcited by 2
- 2024Evaluating the Usability of LLMs in Threat Intelligence Enrichmentcited by 2
- 2025Do Generative AI Tools Ensure Green Code? An Investigative Studycited by 2
- 2026CyberThreat-Eval: Can Large Language Models Automate Real-World Threat Research?cited by 2
- 2025OSINT or BULLSHINT? Exploring Open-Source Intelligence tweets about the Russo-Ukrainian Warcited by 1
- 2025Forecasting Russian Equipment Losses Using Time Series and Deep Learning Modelscited by 1
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2025Hallucination Mitigation using Agentic AI Natural Language-Based Frameworks
- 2025Mitigating Hallucination in Large Language Models (LLMs): An Application-Oriented Survey on RAG, Reasoning, and Agentic Systems
- 2026Cascading Hallucination in Agentic RAG: The CHARM Framework for Detection and Mitigation
- 2026Agentic Artificial Intelligence (AI): Architectures, Taxonomies, and Evaluation of Large Language Model Agents
- 2025LLM-based Agents Suffer from Hallucinations: A Survey of Taxonomy, Methods, and Directions
- 2023LLMs Killed the Script Kiddie: How Agents Supported by Large Language Models Change the Landscape of Network Threat Testing
- 2026SoK: The Attack Surface of Agentic AI - Tools and Autonomy
- 2026Hallucination Cascade: Analyzing Error Propagation in Multi-Agent LLM Systems