mlsanityData to 7 Oct 2026

SkillAggregation: Reference-free LLM-Dependent Aggregation

arXiv 2410.10215 ↗First posted 14 Oct 2024cs.CL cs.LGcited by 6

Not on the map. Why?

Abstract (arXiv)

Large Language Models (LLMs) are increasingly used to assess NLP tasks due to their ability to generate human-like judgments. Single LLMs were used initially, however, recent work suggests using multiple LLMs as judges yields improved performance. An important step in exploiting multiple judgements is the combination stage, aggregation. Existing methods in NLP either assign equal weight to all LLM judgments or are designed for specific tasks such as hallucination detection. This work focuses on aggregating predictions from multiple systems where no reference labels are available. A new method called SkillAggregation is proposed, which learns to combine estimates from LLM judges without needing additional data or ground truth. It extends the Crowdlayer aggregation method, developed for image classification, to exploit the judge estimates during inference. The approach is compared to a range of standard aggregation methods on HaluEval-Dialogue, TruthfulQA and Chatbot Arena tasks. SkillAggregation outperforms Crowdlayer on all tasks, and yields the best performance over all approaches on the majority of tasks.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Builds on 20 of its 39 references that are in the map. First posted on arXiv 14 Oct 2024.

Builds on

20 of 39 references.

References · 20

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2024The Llama 3 Herd of Modelscited by 14,192
  3. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  4. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  5. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
  6. 2020SummEval: Re-evaluating Summarization Evaluationcited by 666
  7. 2023RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedbackcited by 615
  8. 2023Zephyr: Direct Distillation of LM Alignmentcited by 498
  9. 2024From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipelinecited by 484
  10. 2024Replacing Judges with Juries: Evaluating LLM Generations with a Panel of Diverse Modelscited by 301
  11. 2024Aligning with Human Judgement: The Role of Pairwise Preference in Large Language Model Evaluatorscited by 146
  12. 2024Large Language Models are Inconsistent and Biased Evaluatorscited by 146
8 more
  1. 2023PRD: Peer Rank and Discussion Improve Large Language Model based Evaluationscited by 114
  2. 2017Deep learning from crowdscited by 105
  3. 2023LLM Comparative Assessment: Zero-shot NLG Evaluation through Pairwise Comparisons using Large Language Modelscited by 66
  4. 2024Measuring and Reducing LLM Hallucination without Gold-Standard Answerscited by 31
  5. 2018Discourse Coherence in the Wild: A Dataset, Evaluation and Methodscited by 20
  6. 2024CrossCheckGPT: Universal Hallucination Ranking for Multimodal Foundation Modelscited by 18
  7. 2022Knowledge Learning with Crowdsourcing: A Brief Review and Systematic Perspectivecited by 11
  8. 2018A Technical Survey on Statistical Modelling and Design Methods for Crowdsourcing Quality Controlcited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 2; 2026: 4.

6 papers.

  1. 2025Multi-Agent Debate for LLM Judges with Adaptive Stability Detectioncited by 20
  2. 2026Who can we trust? LLM-as-a-jury for Comparative Assessmentcited by 7
  3. 2025Learning an Efficient Multi-Turn Dialogue Evaluator from Multiple LLM Judgescited by 3
  4. 2026The Consensus Trap: Dissecting Subjectivity and the "Ground Truth" Illusion in Data Annotationcited by 1
  5. 2026Enjoy Your Talk: A Human-Centered Benchmark for Multi-Turn Dialogue with Decoupled User Simulation, Target Modeling, and Judgingcited by 1
  6. 2026RAIM: Robust Aggregation of Inexpensive Models for Hallucination Detectioncited by 0