mlsanityData to 14 Oct 2026

Prompting a Weighting Mechanism into LLM-as-a-Judge in Two-Step: A Case Study

arXiv 2502.13396 ↗First posted 19 Feb 2025cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

While Large Language Models (LLMs) have emerged as promising tools for evaluating Natural Language Generation (NLG) tasks, their effectiveness is limited by their inability to appropriately weigh the importance of different topics, often overemphasizing minor details while undervaluing critical information, leading to misleading assessments. Our work proposes an efficient prompt design mechanism to address this specific limitation and provide a case study. Through strategic prompt engineering that incorporates explicit importance weighting mechanisms, we enhance using LLM-as-a-Judge ability to prioritize relevant information effectively, as demonstrated by an average improvement of 6% in the Human Alignment Rate (HAR) metric.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 19 of its 24 references that are in the map. First posted on arXiv 19 Feb 2025.

Builds on

19 of 24 references.

References · 19

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  3. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  4. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,512
  5. 2022Large Language Models are Zero-Shot Reasonerscited by 4,374
  6. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,975
  7. 2019Language Models as Knowledge Bases?cited by 1,936
  8. 2020BLEURT: Learning Robust Metrics for Text Generationcited by 1,203
  9. 2019How Can We Know What Language Models Know?cited by 959
  10. 2023Can Large Language Models Be an Alternative to Human Evaluations?cited by 734
  11. 2021BARTScore: Evaluating Generated Text as Text Generationcited by 722
  12. 2023Preference Ranking Optimization for Human Alignmentcited by 285
7 more
  1. 2023A Survey of Reinforcement Learning from Human Feedbackcited by 264
  2. 2020A Survey of Evaluation Metrics Used for NLG Systemscited by 164
  3. 2024Can LLM be a Personalized Judge?cited by 100
  4. 2023LLMs as Narcissistic Evaluators: When Ego Inflates Evaluation Scorescited by 90
  5. 2023TELeR: A General Taxonomy of LLM Prompts for Benchmarking Complex Taskscited by 49
  6. 2022Toward Human-Like Evaluation for Natural Language Generation with Error Analysiscited by 18
  7. 2024Beyond Labels: Aligning Large Language Models with Human-like Reasoningcited by 6

Built on by

Not cited yet.