mlsanityData to 14 Oct 2026

Vibe Patenting: Evaluating LLM Judges for Professional Patent-Drafting Agents

arXiv 2609.13422 ↗First posted 11 Sep 2026cs.AI cs.LG cs.MAcited by 0

On the map: LLM agents and tool use → Web, search, and research agents

Abstract (arXiv)

LLM judges are increasingly used to evaluate and improve AI-generated outputs, yet their reliability for complex professional work remains unclear. We study this problem through Vibe Patenting, an end-to-end patent-drafting testbed for AI-agent evaluation. A separately-invoked LLM judge evaluates generated patent drafts and provides structured feedback for iterative revision. Across multiple inventions and drafting-agent configurations, judge-guided revision consistently improves judge-assessed quality, while unguided revision tends to saturate. Notably, iterative judge feedback enables a low-reasoning agent to approach the performance of a substantially more expensive high-reasoning agent. Stronger models and increased reasoning generally improve judge-assessed drafting quality, while domain-specific agentic workflows provide further gains. We validate the judge against independent evaluation by a professional patent attorney and find meaningful but strongly metric-dependent agreement and systematic calibration differences. These results highlight both the utility and limitations of LLM judges as evaluators and optimization signals for complex professional workflows.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Web, search, and research agents, one of 16 directions in LLM agents and tool use. Builds on 25 of its 35 references that are in the map. First posted on arXiv 11 Sep 2026.

Builds on

25 of 35 references.

Closest ancestors · 2

Same direction, newest first

  1. 2025Agentic AI for Scientific Discovery: A Survey of Progress, Challenges, and Future Directions12 Mar
  2. 2025Agent Laboratory: Using LLM Agents as Research Assistants8 Jan

Other references · 23

  1. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
  2. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
  3. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,276
  4. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,442
  5. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,211
  6. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,276
  7. 2023AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversationcited by 1,988
  8. 2024Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameterscited by 1,986
  9. 2023MetaGPT: Meta Programming for A Multi-Agent Collaborative Frameworkcited by 1,808
  10. 2024A Survey on LLM-as-a-Judgecited by 1,423
  11. 2023ChemCrow: Augmenting large-language models with chemistry toolscited by 631
  12. 2023Prometheus: Inducing Fine-grained Evaluation Capability in Language Modelscited by 575
11 more
  1. 2024From Generation to Judgment: Opportunities and Challenges of LLM-as-a-judgecited by 494
  2. 2025Accelerating scientific discovery with Co-Scientistcited by 401
  3. 2024Replacing Judges with Juries: Evaluating LLM Generations with a Panel of Diverse Modelscited by 326
  4. 2024JudgeBench: A Benchmark for Evaluating LLM-based Judgescited by 318
  5. 2023Evaluating Large Language Models at Evaluating Instruction Followingcited by 302
  6. 2024Self-Preference Bias in LLM-as-a-Judgecited by 258
  7. 2023Generative Judge for Evaluating Alignmentcited by 169
  8. 2026Towards End-to-End Automation of AI Researchcited by 159
  9. 2024Can Large Language Models Generate High-quality Patent Claims?cited by 19
  10. 2025PatentScore: Multi-dimensional Evaluation of LLM-Generated Patent Claimscited by 11
  11. 2025Towards Better Evaluation for Generated Patent Claimscited by 8

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025PATENTWRITER: A Benchmarking Study for Patent Drafting with LLMs
  2. 2026Agent-as-a-Judge
  3. 2026Time to REFLECT: Can We Trust LLM Judges for Evidence-based Research Agents?
  4. 2025When AIs Judge AIs: The Rise of Agent-as-a-Judge Evaluation for LLMs
  5. 2026JudgeFlow: Agentic Workflow Optimization via Block Judge
  6. 2025AutoSpec: An Agentic Framework for Automatically Drafting Patent Specification
  7. 2026AgentJudgeBench: A Multi-Difficulty Benchmark for Evaluating LLM Judges on Agentic Tool-Calling
  8. 2025Generate, Evaluate, Iterate: Synthetic Data for Human-in-the-Loop Refinement of LLM Judges