# Language Model Council: Democratically Benchmarking Foundation Models on Highly Subjective Tasks

arXiv: https://arxiv.org/abs/2406.08598 · HTML: https://mlsanity.com/paper/2406.08598/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 12 Jun 2024.

## Built on by (most cited first, 18 of 18)

- [Beyond Majority Voting: LLM Aggregation by Leveraging Higher-Order Information](https://mlsanity.com/paper/2510.01499/) (2025)
- [Self-Preference Bias in Rubric-Based Evaluation of Large Language Models](https://mlsanity.com/paper/2604.06996/) (2026)
- [Conversational User-AI Intervention: A Study on Prompt Rewriting for Improved LLM Response Generation](https://mlsanity.com/paper/2503.16789/) (2025)
- [Don't Throw Away Your Pretrained Model](https://mlsanity.com/paper/2510.09913/) (2025)
- [The Pluralistic Moral Gap: Understanding Judgment and Value Differences between Humans and Large Language Models](https://mlsanity.com/paper/2507.17216/) (2025)
- [Who can we trust? LLM-as-a-jury for Comparative Assessment](https://mlsanity.com/paper/2602.16610/) (2026)
- [The Single-Multi Evolution Loop for Self-Improving Model Collaboration Systems](https://mlsanity.com/paper/2602.05182/) (2026)
- [Council Mode: A Heterogeneous Multi-Agent Consensus Framework for Reducing LLM Hallucination and Bias](https://mlsanity.com/paper/2604.02923/) (2026)
- [When Does Delegation Beat Majority? A Delegation-Based Aggregator for Multi-Sample LLM Inference](https://mlsanity.com/paper/2606.08098/) (2026)
- [Sigma-Mem: An Online Reliability Memory for LLM-based Multi-Agent Systems](https://mlsanity.com/paper/2607.27958/) (2026)
- [Jagged Judges: Epistemic Stability Under Perturbation, Pressure, and Persistence](https://mlsanity.com/paper/2608.12645/) (2026)
- [Vibe Coding on Trial: Operating Characteristics of Unanimous LLM Juries](https://mlsanity.com/paper/2602.18492/) (2026)
- [Certifiable Semantic Agreement Among LLM Agents: What the Admissibility Instrument Decides](https://mlsanity.com/paper/2606.07316/) (2026)
- [Multi-Expert Conformal Risk Control for Pairwise LLM Judging in Open-Ended Dialogue](https://mlsanity.com/paper/2608.26529/) (2026)
- [What Does Multi-Agent LLM Debate Actually Change? A Layered Analysis of Disagreement and Answer Quality](https://mlsanity.com/paper/2609.08016/) (2026)
- [You're Hired: Strategic Model Selection for LLM Collaboration](https://mlsanity.com/paper/2609.38816/) (2026)
- [RAIM: Robust Aggregation of Inexpensive Models for Hallucination Detection](https://mlsanity.com/paper/2609.39229/) (2026)
- [Understanding Machine Taste: Aesthetic Agreement Across Foundation Models in K-pop](https://mlsanity.com/paper/2610.17731/) (2026)

## Abstract (arXiv)

As Large Language Models (LLMs) continue to evolve, evaluating them remains a persistent challenge. Many recent evaluations use LLMs as judges to score outputs from other LLMs, often relying on a single large model like GPT-4o. However, using a single LLM judge is prone to intra-model bias, and many tasks - such as those related to emotional intelligence, creative writing, and persuasiveness - may be too subjective for a single model to judge fairly. We introduce the Language Model Council (LMC), where a group of LLMs collaborate to create tests, respond to them, and evaluate each other's responses to produce a ranking in a democratic fashion. Unlike previous approaches that focus on reducing cost or bias by using a panel of smaller models, our work examines the benefits and nuances of a fully inclusive LLM evaluation system. In a detailed case study on emotional intelligence, we deploy a council of 20 recent LLMs to rank each other on open-ended responses to interpersonal conflicts. Our results show that the LMC produces rankings that are more separable and more robust, and through a user study, we show that they are more consistent with human evaluations than any individual LLM judge. Using all LLMs for judging can be costly, however, so we use Monte Carlo simulations and hand-curated sub-councils to study hypothetical council compositions and discuss the value of the incremental LLM judge.