mlsanityData to 14 Oct 2026

ALLURE: Auditing and Improving LLM-based Evaluation of Text using Iterative In-Context-Learning

arXiv 2309.13701 ↗First posted 24 Sep 2023cs.CL cs.AI cs.HCcited by 15

Not on the map. Why?

Abstract (arXiv)

From grading papers to summarizing medical documents, large language models (LLMs) are evermore used for evaluation of text generated by humans and AI alike. However, despite their extensive utility, LLMs exhibit distinct failure modes, necessitating a thorough audit and improvement of their text evaluation capabilities. Here we introduce ALLURE, a systematic approach to Auditing Large Language Models Understanding and Reasoning Errors. ALLURE involves comparing LLM-generated evaluations with annotated data, and iteratively incorporating instances of significant deviation into the evaluator, which leverages in-context learning (ICL) to enhance and improve robust evaluation of text by LLMs. Through this iterative process, we refine the performance of the evaluator LLM, ultimately reducing reliance on human annotators in the evaluation process. We anticipate ALLURE to serve diverse applications of LLMs in various domains related to evaluation of textual data, such as medical summarization, education, and and productivity.

Key facts. Cited by 15 papers in the map (data through 14 Oct 2026). Builds on 41 of its 66 references that are in the map. First posted on arXiv 24 Sep 2023.

Builds on

41 of 66 references.

References · 41

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2022Training language models to follow instructions with human feedbackcited by 15,971
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  5. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  6. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  7. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,512
  8. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 3,104
  9. 2023A Survey of Large Language Modelscited by 2,576
  10. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,276
  11. 2022Large Language Models Encode Clinical Knowledgecited by 1,830
  12. 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,392
29 more
  1. 2022LaMDA: Language Models for Dialog Applicationscited by 1,220
  2. 2021What Makes Good In-Context Examples for GPT-3?cited by 1,195
  3. 2023Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Promptingcited by 1,122
  4. 2019PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarizationcited by 1,102
  5. 2023Large Language Models are not Fair Evaluatorscited by 1,085
  6. 2015Teaching Machines to Read and Comprehendcited by 1,049
  7. 2021BARTScore: Evaluating Generated Text as Text Generationcited by 722
  8. 2022A Survey on In-context Learningcited by 721
  9. 2020SummEval: Re-evaluating Summarization Evaluationcited by 674
  10. 2023Is ChatGPT a Good NLG Evaluator? A Preliminary Studycited by 422
  11. 2023GPTScore: Evaluate as You Desirecited by 361
  12. 2023Causal Reasoning and Large Language Models: Opening a New Frontier for Causalitycited by 353
  13. 2022Selective Annotation Makes Language Models Better Few-Shot Learnerscited by 243
  14. 2022Why Can GPT Learn In-Context? Language Models Implicitly Perform Gradient Descent as Meta-Optimizerscited by 204
  15. 2023PRD: Peer Rank and Discussion Improve Large Language Model based Evaluationscited by 114
  16. 2016Model-Free Episodic Controlcited by 102
  17. 2023Red teaming ChatGPT via Jailbreaking: Bias, Robustness, Reliability and Toxicitycited by 89
  18. 2023Practical and Ethical Challenges of Large Language Models in Education: A Systematic Scoping Reviewcited by 83
  19. 2023On Learning to Summarize with Large Language Models as Referencescited by 80
  20. 2019Reinforcement Learning for Temporal Logic Control Synthesis with Probabilistic Satisfaction Guaranteescited by 77
  21. 2018Logically-Constrained Reinforcement Learningcited by 51
  22. 2022On the Blind Spots of Model-Based Evaluation Metrics for Text Generationcited by 49
  23. 2020Cautious Reinforcement Learning with Logical Constraintscited by 45
  24. 2023How Ready are Pre-trained Abstractive Models and LLMs for Legal Case Judgement Summarization?cited by 43
  25. 2019Certified Reinforcement Learning with Logic Guidancecited by 36
  26. 2018Improving Abstraction in Text Summarizationcited by 35
  27. 2018Logically-Constrained Neural Fitted Q-Iterationcited by 17
  28. 2022A Survey on Medical Document Summarizationcited by 14
  29. 2022LCRL: Certified Policy Synthesis via Logically-Constrained Reinforcement Learningcited by 11

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 1; 2024: 11; 2025: 1; 2026: 2.

15 papers.

Sort:

By area:

  1. 2024From Generation to Judgment: Opportunities and Challenges of LLM-as-a-judgecited by 494
  2. 2024LLM-based NLG Evaluation: Current Status and Challengescited by 73
  3. 2024Finding Blind Spots in Evaluator LLMs with Interpretable Checklistscited by 32
  4. 2024Navigating LLM Ethics: Advancements, Challenges, and Future Directionscited by 28
  5. 2024Grade Like a Human: Rethinking Automated Assessment with Large Language Modelscited by 16
  6. 2024Could ChatGPT get an Engineering Degree? Evaluating Higher Education Vulnerability to AI Assistantscited by 10
  7. 2024LLMAuditor: A Framework for Auditing Large Language Models Using Human-in-the-Loopcited by 8
  8. 2023Decoding In-Context Learning: Neuroscience-inspired Analysis of Representations in Large Language Modelscited by 7
  9. 2024Leveraging the Power of LLMs: A Fine-Tuning Approach for High-Quality Aspect-Based Summarizationcited by 3
  10. 2024Aligning Large Language Models from Self-Reference AI Feedback with one General Principlecited by 2
  11. 2024From Cool Demos to Production-Ready FMware: Core Challenges and a Technology Roadmapcited by 2
  12. 2025Comparing LLM Text Annotation Skills: A Study on Human Rights Violations in Social Media Datacited by 1
3 more
  1. 2024Intelligent Monitoring Framework for Cloud Services: A Data-Driven Approachcited by 0
  2. 2026DA-RAC: Distance-Aware Calibration of LLM Judges for Trustworthy AI Auditingcited by 0
  3. 2026Do Social Patterns Hold in Synthetic Data? Analyzing Cyberbullying Dynamics in LLM-Generated and Authentic Dialoguescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Exploring the Reliability of Large Language Models as Customized Evaluators for Diverse NLP Tasks
  2. 2024Can Large Language Models be Trusted for Evaluation? Scalable Meta-Evaluation of LLMs as Evaluators via Agent Debate
  3. 2024DnA-Eval: Enhancing Large Language Model Evaluation through Decomposition and Aggregation
  4. 2023Through the Lens of Core Competency: Survey on Evaluation of Large Language Models
  5. 2026A primer on evaluation methods for large language models in healthcare
  6. 2025Learned-Rule-Augmented Large Language Model Evaluators
  7. 2025Training an LLM-as-a-Judge Model: Pipeline, Insights, and Practical Lessons
  8. 2025Analysis of instruction-based LLMs' capabilities to score and judge text-input problems in an academic setting