# A Better LLM Evaluator for Text Generation: The Impact of Prompt Output Sequencing and Optimization

arXiv: https://arxiv.org/abs/2406.09972 · HTML: https://mlsanity.com/paper/2406.09972/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 9 of its 15 references that are in the map.
- First posted on arXiv 14 Jun 2024.

## Builds on (most cited first, 9 of 9)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Generative Agents: Interactive Simulacra of Human Behavior](https://mlsanity.com/paper/2304.03442/) (2023)
- [G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment](https://mlsanity.com/paper/2303.16634/) (2023)
- [Large Language Models as Optimizers](https://mlsanity.com/paper/2309.03409/) (2023)
- [GrIPS: Gradient-free, Edit-based Instruction Search for Prompting Large Language Models](https://mlsanity.com/paper/2203.07281/) (2022)
- [Large Language Models are Not Yet Human-Level Evaluators for Abstractive Summarization](https://mlsanity.com/paper/2305.13091/) (2023)
- [InstructZero: Efficient Instruction Optimization for Black-Box Large Language Models](https://mlsanity.com/paper/2306.03082/) (2023)
- [The language of prompting: What linguistic properties make a prompt successful?](https://mlsanity.com/paper/2311.01967/) (2023)

## Built on by (most cited first, 11 of 11)

- [MemSim: A Bayesian Simulator for Evaluating Memory of LLM-based Personal Assistants](https://mlsanity.com/paper/2409.20163/) (2024)
- [Any Large Language Model Can Be a Reliable Judge: Debiasing with a Reasoning-based Bias Detector](https://mlsanity.com/paper/2505.17100/) (2025)
- [PersoBench: Benchmarking Personalized Response Generation in Large Language Models](https://mlsanity.com/paper/2410.03198/) (2024)
- [A review of faithfulness metrics for hallucination assessment in Large Language Models](https://mlsanity.com/paper/2501.00269/) (2024)
- [Decision Information Meets Large Language Models: The Future of Explainable Operations Research](https://mlsanity.com/paper/2502.09994/) (2025)
- [Solar Photovoltaic Assessment with Large Language Model](https://mlsanity.com/paper/2507.19144/) (2025)
- [SAGED: A Holistic Bias-Benchmarking Pipeline for Language Models with Customisable Fairness Calibration](https://mlsanity.com/paper/2409.11149/) (2024)
- [HPSS: Heuristic Prompting Strategy Search for LLM Evaluators](https://mlsanity.com/paper/2502.13031/) (2025)
- [Image First or Text First? Optimising the Sequencing of Modalities in Large Language Model Prompting and Reasoning Tasks](https://mlsanity.com/paper/2410.03062/) (2024)
- [Difficult Task Yes but Simple Task No: Unveiling the Laziness in Multimodal LLMs](https://mlsanity.com/paper/2410.11437/) (2024)
- [DSGram: Dynamic Weighting Sub-Metrics for Grammatical Error Correction in the Era of Large Language Models](https://mlsanity.com/paper/2412.12832/) (2024)

## Abstract (arXiv)

This research investigates prompt designs of evaluating generated texts using large language models (LLMs). While LLMs are increasingly used for scoring various inputs, creating effective prompts for open-ended text evaluation remains challenging due to model sensitivity and subjectivity in evaluation of text generation. Our study experimented with different prompt structures, altering the sequence of output instructions and including explanatory reasons. We found that the order of presenting reasons and scores significantly influences LLMs' scoring, with a different level of rule understanding in the prompt. An additional optimization may enhance scoring alignment if sufficient data is available. This insight is crucial for improving the accuracy and consistency of LLM-based evaluations.