mlsanityData to 7 Oct 2026

A Better LLM Evaluator for Text Generation: The Impact of Prompt Output Sequencing and Optimization

arXiv 2406.09972 ↗First posted 14 Jun 2024cs.CLcited by 11

Not on the map. Why?

Abstract (arXiv)

This research investigates prompt designs of evaluating generated texts using large language models (LLMs). While LLMs are increasingly used for scoring various inputs, creating effective prompts for open-ended text evaluation remains challenging due to model sensitivity and subjectivity in evaluation of text generation. Our study experimented with different prompt structures, altering the sequence of output instructions and including explanatory reasons. We found that the order of presenting reasons and scores significantly influences LLMs' scoring, with a different level of rule understanding in the prompt. An additional optimization may enhance scoring alignment if sufficient data is available. This insight is crucial for improving the accuracy and consistency of LLM-based evaluations.

Key facts. Cited by 11 papers in the map (data through 7 Oct 2026). Builds on 9 of its 15 references that are in the map. First posted on arXiv 14 Jun 2024.

Builds on

9 of 15 references.

References · 9

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  3. 2023Generative Agents: Interactive Simulacra of Human Behaviorcited by 3,500
  4. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,205
  5. 2023Large Language Models as Optimizerscited by 830
  6. 2022GrIPS: Gradient-free, Edit-based Instruction Search for Prompting Large Language Modelscited by 127
  7. 2023Large Language Models are Not Yet Human-Level Evaluators for Abstractive Summarizationcited by 70
  8. 2023InstructZero: Efficient Instruction Optimization for Black-Box Large Language Modelscited by 68
  9. 2023The language of prompting: What linguistic properties make a prompt successful?cited by 43

Built on by

Cited by year

Citing papers per year: 2024: 7; 2025: 4.

11 papers.

  1. 2024MemSim: A Bayesian Simulator for Evaluating Memory of LLM-based Personal Assistantscited by 24
  2. 2025Any Large Language Model Can Be a Reliable Judge: Debiasing with a Reasoning-based Bias Detectorcited by 14
  3. 2024PersoBench: Benchmarking Personalized Response Generation in Large Language Modelscited by 12
  4. 2024A review of faithfulness metrics for hallucination assessment in Large Language Modelscited by 9
  5. 2025Decision Information Meets Large Language Models: The Future of Explainable Operations Researchcited by 9
  6. 2025Solar Photovoltaic Assessment with Large Language Modelcited by 7
  7. 2024SAGED: A Holistic Bias-Benchmarking Pipeline for Language Models with Customisable Fairness Calibrationcited by 6
  8. 2025HPSS: Heuristic Prompting Strategy Search for LLM Evaluatorscited by 5
  9. 2024Difficult Task Yes but Simple Task No: Unveiling the Laziness in Multimodal LLMscited by 3
  10. 2024Image First or Text First? Optimising the Sequencing of Modalities in Large Language Model Prompting and Reasoning Taskscited by 2
  11. 2024DSGram: Dynamic Weighting Sub-Metrics for Grammatical Error Correction in the Era of Large Language Modelscited by 1