# From Reviewers' Lens: Understanding Bug Bounty Report Invalid Reasons with LLMs

arXiv: https://arxiv.org/abs/2511.18608 · HTML: https://mlsanity.com/paper/2511.18608/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Safety alignment, refusal and guardrails, one of 12 directions in LLM safety and security.
- Builds on 7 of its 35 references that are in the map.
- First posted on arXiv 23 Nov 2025.

## Placed in

- LLM safety and security / [Safety alignment, refusal and guardrails](https://mlsanity.com/safety/safety-alignment-refusal-and-guardrails/)

## Builds on (most cited first, 7 of 7)

- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Equality of Opportunity in Supervised Learning](https://mlsanity.com/paper/1610.02413/) (2016)
- [Bug Hunters' Perspectives on the Challenges and Benefits of the Bug Bounty Ecosystem](https://mlsanity.com/paper/2301.04781/) (2023)
- [Given Enough Eyeballs, All Bugs Are Shallow? Revisiting Eric Raymond with Bug Bounty Programs](https://mlsanity.com/paper/1608.03445/) (2016)
- [A Mixed-Methods Study of Open-Source Software Maintainers On Vulnerability Management and Platform Security Features](https://mlsanity.com/paper/2409.07669/) (2024)
- [A Deep Dive Into How Open-Source Project Maintainers Review and Resolve Bug Bounty Reports](https://mlsanity.com/paper/2409.07670/) (2024)
- [Investigating Vulnerability Disclosures in Open-Source Software Using Bug Bounty Reports and Security Advisories](https://mlsanity.com/paper/2501.17748/) (2025)

## Abstract (arXiv)

Bug bounty platforms (e.g., HackerOne, BugCrowd) leverage crowd-sourced vulnerability discovery to improve continuous coverage, reduce the cost of discovery, and serve as an integral complement to internal red teams. With the rise of AI-generated bug reports, little work exists to help bug hunters understand why these reports are labeled as invalid. To improve report quality and reduce reviewers' burden, it is critical to predict invalid reports and interpret invalid reasons. In this work, we conduct an empirical study with the purpose of helping bug hunters understand the validity of reports. We collect a dataset of 9,942 disclosed bug bounty reports, including 1,400 invalid reports, and evaluate whether state-of-the-art large language models can identify invalid reports. While models such as GPT-5, DeepSeek, and a fine-tuned RoBERTa achieve strong overall accuracy, they consistently struggle to detect invalid cases, showing a tendency to over-accept reports. To improve invalidity detection, we build a taxonomy of rejection reasons for Information Disclosure vulnerabilities and incorporate it into a retrieval-augmented generation (RAG) framework. This approach substantially improves classification consistency and reduces bias. We also examine whether reviewer decisions may be influenced by factors beyond the content of the report. Our analysis shows that reporters with higher reputations tend to receive more favorable outcomes in borderline cases, suggesting that perceived expertise can influence review judgments. Overall, our findings highlight the challenges of invalid report identification and show that combining LLMs with structured reviewer knowledge can support more transparent and consistent vulnerability report review.