mlsanityData to 14 Oct 2026

GLIDER: Grading LLM Interactions and Decisions using Explainable Ranking

arXiv 2412.14140 ↗First posted 18 Dec 2024cs.CL cs.AIcited by 14

Not on the map. Why?

Abstract (arXiv)

The LLM-as-judge paradigm is increasingly being adopted for automated evaluation of model outputs. While LLM judges have shown promise on constrained evaluation tasks, closed source LLMs display critical shortcomings when deployed in real world applications due to challenges of fine grained metrics and explainability, while task specific evaluation models lack cross-domain generalization. We introduce GLIDER, a powerful 3B evaluator LLM that can score any text input and associated context on arbitrary user defined criteria. GLIDER shows higher Pearson's correlation than GPT-4o on FLASK and greatly outperforms prior evaluation models, achieving comparable performance to LLMs 17x its size. GLIDER supports fine-grained scoring, multilingual reasoning, span highlighting and was trained on 685 domains and 183 criteria. Extensive qualitative analysis shows that GLIDER scores are highly correlated with human judgments, with 91.3% human agreement. We have open-sourced GLIDER to facilitate future research.

Key facts. Cited by 14 papers in the map (data through 14 Oct 2026). Builds on 32 of its 53 references that are in the map. First posted on arXiv 18 Dec 2024.

Builds on

32 of 53 references.

References · 32

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  3. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,512
  4. 2023ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIscited by 1,947
  5. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,917
  6. 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,055
  7. 2020COMET: A Neural Framework for MT Evaluationcited by 1,033
  8. 2020SummEval: Re-evaluating Summarization Evaluationcited by 674
  9. 2023PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallelcited by 660
  10. 2023RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedbackcited by 625
  11. 2024Prometheus 2: An Open Source Language Model Specialized in Evaluating Other Language Modelscited by 446
  12. 2024Scaling Synthetic Data Creation with 1,000,000,000 Personascited by 417
20 more
  1. 2023JudgeLM: Fine-tuned Large Language Models are Scalable Judgescited by 367
  2. 2020Evaluation of Text Generation: A Surveycited by 277
  3. 2024LiveBench: A Challenging, Contamination-Limited LLM Benchmarkcited by 262
  4. 2024LLMs instead of Human Judges? A Large Scale Empirical Study across 20 NLP Evaluation Taskscited by 243
  5. 2024BABILong: Testing the Limits of LLMs with Long Context Reasoning-in-a-Haystackcited by 215
  6. 2023FLASK: Fine-grained Language Model Evaluation based on Alignment Skill Setscited by 201
  7. 2024Can LLM be a Personalized Judge?cited by 100
  8. 2024CheckEval: A reliable LLM-as-a-Judge framework for evaluating text generation using checklistscited by 94
  9. 2024The BiGGen Bench: A Principled Benchmark for Fine-grained Evaluation of Language Models with Language Modelscited by 93
  10. 2023Do Models Explain Themselves? Counterfactual Simulatability of Natural Language Explanationscited by 85
  11. 2024Foundational Autoraters: Taming Large Language Models for Better Automatic Evaluationcited by 74
  12. 2024LLM-based NLG Evaluation: Current Status and Challengescited by 73
  13. 2024M-RewardBench: Evaluating Reward Models in Multilingual Settingscited by 50
  14. 2024Anchored Preference Optimization and Contrastive Revisions: Addressing Underspecification in Alignmentcited by 46
  15. 2024Direct Judgement Preference Optimizationcited by 33
  16. 2024LLM-as-a-Judge & Reward Model: What They Can and Cannot Docited by 32
  17. 2024Spotting AI's Touch: Identifying LLM-Paraphrased Spans in Textcited by 16
  18. 2023EIPE-text: Evaluation-Guided Iterative Plan Extraction for Long-Form Narrative Text Generationcited by 12
  19. 2024Coarse-to-Fine Highlighting: Reducing Knowledge Hallucination in Large Language Modelscited by 7
  20. 2019Towards a Metric for Automated Conversational Dialogue System Evaluation and Improvementcited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 8; 2026: 6.

14 papers.

Sort:

By area:

  1. 2025A Survey of Frontiers in LLM Reasoning: Inference Scaling, Learning to Reason, and Agentic Systemscited by 111
  2. 2025TRAIL: Trace Reasoning and Agentic Issue Localizationcited by 104
  3. 2025Evaluating Judges as Evaluators: The JETTS Benchmark of LLM-as-Judges as Test-Time Scaling Evaluatorscited by 44
  4. 2025Does Context Matter? ContextualJudgeBench for Evaluating LLM-based Judges in Contextual Settingscited by 24
  5. 2026Generating and Refining Dynamic Evaluation Rubrics for LLM-as-a-Judgecited by 8
  6. 2025FRED: Financial Retrieval-Enhanced Detection and Editing of Hallucinations in Language Modelscited by 6
  7. 2026CompliBench: Benchmarking LLM Judges for Compliance Violation Detection in Dialogue Systemscited by 6
  8. 2026MARCH: Multi-Agent Reinforced Self-Check for LLM Hallucinationcited by 5
  9. 2026PReMISE: Policy Rubrics as Measurement Specifications for LLM Judgescited by 4
  10. 2025Multi-Domain Explainability of Preferencescited by 2
  11. 2025Neural Models and Language Model Prompting for the Multidimensional Evaluation of Open-Ended Conversationscited by 2
  12. 2025SAS-Bench: A Fine-Grained Benchmark for Evaluating Short Answer Scoring with Large Language Modelscited by 1
2 more
  1. 2026BARRED: Synthetic Training of Custom Policy Guardrails via Asymmetric Debatecited by 0
  2. 2026Beyond Correctness: Validity-Oriented Evaluation of Biomedical LLM Judgescited by 0