mlsanityData to 14 Oct 2026

JavaVulBench: A Java Vulnerability Benchmark with Realistic Splits, a Unified Multi-Backend Harness, and a Leakage-Aware Evaluation Mode

arXiv 2607.02825 ↗First posted 2 Jul 2026cs.CR cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

We release textscJavaVulBench, a benchmark dataset and evaluation harness for Java vulnerability detection. The dataset contains ~30,600 Java methods spanning 1,740 CVEs and 700+ projects, labelled at both method and line granularity, with per-CVE publication dates and five realistic split strategies: random, project-disjoint, temporal, deduplicated, and unseen CWE-family. The harness provides a single textttLlmPrediction schema across three backend families (encoder classifiers, local generative models served by Ollama, and API-served LLMs routed through OpenRouter) so that twelve reference detectors CodeBERT, GraphCodeBERT, UniXcoder, DeepSeek-Coder-1.3B, and eight API/open-weight LLMs (GPT-4o, GPT-4.1-mini, Claude Sonnet~4, DeepSeek-v3, DeepSeek-Coder-v2, Qwen-2.5-Coder-14B/7B, CodeLlama-13B) are evaluated under identical conditions from a single command. A pre-training contamination audit is shipped alongside every model so users can separate genuinely unseen test CVEs from potentially memorised ones. Data, code, and fine-tuned checkpoints are archived on Zenodo [31] and short demonstration video is available on YouTube (https://www.youtube.com/watch?v=nMTXhqkuoM) https://www.youtube.com/watch?v=nMTXhqkuoM.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 13 of its 33 references that are in the map. First posted on arXiv 2 Jul 2026.

Builds on

13 of 33 references.

References · 13

  1. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  2. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  3. 2017On Calibration of Modern Neural Networkscited by 4,960
  4. 2017Billion-scale similarity search with GPUscited by 2,497
  5. 2023Code Llama: Open Foundation Models for Codecited by 2,139
  6. 2020CodeBERT: A Pre-Trained Model for Programming and Natural Languagescited by 1,310
  7. 2020GraphCodeBERT: Pre-training Code Representations with Data Flowcited by 609
  8. 2022UniXcoder: Unified Cross-Modal Pre-training for Code Representationcited by 361
  9. 2024DeepSeek-Coder-V2: Breaking the Barrier of Closed-Source Models in Code Intelligencecited by 331
  10. 2019Devign: Effective Vulnerability Identification by Learning Comprehensive Program Semantics via Graph Neural Networkscited by 272
  11. 2021CVEfixes: Automated Collection of Vulnerabilities and Their Fixes from Open-Source Softwarecited by 91
  12. 2023How Effective Are Neural Networks for Fixing Security Vulnerabilitiescited by 52
1 more
  1. 2024Pre-training by Predicting Program Dependencies for Vulnerability Analysis Taskscited by 17

Built on by

Not cited yet.