# LLM4VV: Exploring LLM-as-a-Judge for Validation and Verification Testsuites

arXiv: https://arxiv.org/abs/2408.11729 · HTML: https://mlsanity.com/paper/2408.11729/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 21 Aug 2024.

## Built on by (most cited first, 4 of 4)

- [MARCO: Multi-Agent Code Optimization with Real-Time Knowledge Integration for High-Performance Computing](https://mlsanity.com/paper/2505.03906/) (2025)
- [Toward Automated Validation of Language Model Synthesized Test Cases using Semantic Entropy](https://mlsanity.com/paper/2411.08254/) (2024)
- [Overcoming Vision Language Model Challenges in Diagram Understanding: A Proof-of-Concept with XML-Driven Large Language Models Solutions](https://mlsanity.com/paper/2502.04389/) (2025)
- [GeoRisk-RAG: A Hierarchy-Aware Risk Framework for Improving RAG Reliability through Selective Answering](https://mlsanity.com/paper/2608.22634/) (2026)

## Abstract (arXiv)

Large Language Models (LLM) are evolving and have significantly revolutionized the landscape of software development. If used well, they can significantly accelerate the software development cycle. At the same time, the community is very cautious of the models being trained on biased or sensitive data, which can lead to biased outputs along with the inadvertent release of confidential information. Additionally, the carbon footprints and the un-explainability of these black box models continue to raise questions about the usability of LLMs. With the abundance of opportunities LLMs have to offer, this paper explores the idea of judging tests used to evaluate compiler implementations of directive-based programming models as well as probe into the black box of LLMs. Based on our results, utilizing an agent-based prompting approach and setting up a validation pipeline structure drastically increased the quality of DeepSeek Coder, the LLM chosen for the evaluation purposes.