mlsanityData to 14 Oct 2026

Time To Impeach LLM-as-a-Judge: Programs are the Future of Evaluation

arXiv 2506.10403 ↗First posted 12 Jun 2025cs.LG cs.AIcited by 6

Not on the map. Why?

Abstract (arXiv)

Large language models (LLMs) are widely used to evaluate the quality of LLM generations and responses, but this leads to significant challenges: high API costs, uncertain reliability, inflexible pipelines, and inherent biases. To address these, we introduce PAJAMA (Program-As-a-Judge for Automated Model Assessment), a new alternative that uses LLMs to synthesize executable judging programs instead of directly scoring responses. These synthesized programs can be stored and run locally, costing orders of magnitude less while providing interpretable, and auditable judging logic that can be easily adapted. Program-based judges mitigate biases, improving judgment consistency by 15.83% and reducing biased responses by 23.7% on average compared to a Qwen2.5-14B-based LLM-as-a-judge. When program judgments are distilled into a model, PAJAMA outperforms LLM-as-a-judge on the challenging CHAT-HARD subset of RewardBench, outperforming metrics by 2.19% on Prometheus and 8.67% on the JudgeLM dataset, all at three orders of magnitude lower cost.

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Builds on 29 of its 38 references that are in the map. First posted on arXiv 12 Jun 2025.

Builds on

29 of 38 references.

References · 29

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  3. 2024GPT-4o System Cardcited by 4,588
  4. 2017Deep reinforcement learning from human preferencescited by 3,912
  5. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,998
  6. 2024Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preferencecited by 1,184
  7. 2024M3-Embedding: Multi-Linguality, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillationcited by 1,073
  8. 2023Prometheus: Inducing Fine-grained Evaluation Capability in Language Modelscited by 575
  9. 2024RewardBench: Evaluating Reward Models for Language Modelingcited by 499
  10. 2024LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation Methodscited by 465
  11. 2017Snorkel: Rapid Training Data Creation with Weak Supervisioncited by 372
  12. 2024Justice or Prejudice? Quantifying Biases in LLM-as-a-Judgecited by 368
17 more
  1. 2023JudgeLM: Fine-tuned Large Language Models are Scalable Judgescited by 367
  2. 2024Humans or LLMs as the Judge? A Study on Judgement Biasescited by 323
  3. 2023PandaLM: An Automatic Evaluation Benchmark for LLM Instruction Tuning Optimizationcited by 316
  4. 2016Data Programming: Creating Large Training Sets, Quicklycited by 246
  5. 2024Agent-as-a-Judge: Evaluate Agents with Agentscited by 238
  6. 2020Learning from the Worst: Dynamically Generated Datasets to Improve Online Hate Detectioncited by 221
  7. 2024QuRating: Selecting High-Quality Data for Training Language Modelscited by 157
  8. 2018Training Complex Models with Multi-Task Weak Supervisioncited by 106
  9. 2020Fast and Three-rious: Speeding Up Weak Supervision with Triplet Methodscited by 98
  10. 2024Systematic Evaluation of LLM-as-a-Judge in LLM Alignment Tasks: Explainable Metrics and Diverse Prompt Templatescited by 72
  11. 2023FactKB: Generalizable Factuality Evaluation using Language Models Enhanced with Factual Knowledgecited by 55
  12. 2024Weak-to-Strong Generalization Through the Data-Centric Lenscited by 21
  13. 2024Discovering Bias in Latent Space: An Unsupervised Debiasing Approachcited by 15
  14. 2022Lifting Weak Supervision To Structured Predictioncited by 10
  15. 2024The ALCHEmist: Automated Labeling 500x CHEaper Than LLM Data Annotatorscited by 10
  16. 2022AutoWS-Bench-101: Benchmarking Automated Weak Supervision with 100 Labelscited by 7
  17. 2025ScriptoriumWS: A Code Generation Assistant for Weak Supervisioncited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 5.

6 papers.

  1. 2026CARE: Confounder-Aware Aggregation for Reliable LLM Evaluationcited by 13
  2. 2025Automating Benchmark Designcited by 3
  3. 2026Beyond LLM-as-a-Judge: Deterministic Metrics for Multilingual Generative Text Evaluationcited by 3
  4. 2026Multilingual Prompt Localization for Agent-as-a-Judge: Language and Backbone Sensitivity in Requirement-Level Evaluationcited by 2
  5. 2026TREK: A Travel Reasoning and Evaluation Kit for LLM Agents in Complex Trip Planningcited by 2
  6. 2026JudgeMoE: Distributional Aggregation for LLM-as-a-Judgecited by 0