# Towards Multi-Agent-Simulation-Based Community Note Evaluation

arXiv: https://arxiv.org/abs/2606.18268 · HTML: https://mlsanity.com/paper/2606.18268/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 19 of its 35 references that are in the map.
- First posted on arXiv 3 Jun 2026.

## Builds on (most cited first, 19 of 19)

- [Generative Agents: Interactive Simulacra of Human Behavior](https://mlsanity.com/paper/2304.03442/) (2023)
- [G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment](https://mlsanity.com/paper/2303.16634/) (2023)
- [A Survey on Automated Fact-Checking](https://mlsanity.com/paper/2108.11896/) (2021)
- [FLASK: Fine-grained Language Model Evaluation based on Alignment Skill Sets](https://mlsanity.com/paper/2307.10928/) (2023)
- [Factcheck-Bench: Fine-Grained Evaluation Benchmark for Automatic Fact-checkers](https://mlsanity.com/paper/2311.09000/) (2023)
- [Show Me the Work: Fact-Checkers' Requirements for Explainable Automated Fact-Checking](https://mlsanity.com/paper/2502.09083/) (2025)
- [Supernotes: Driving Consensus in Crowd-Sourced Fact-Checking](https://mlsanity.com/paper/2411.06116/) (2024)
- [Human and technological infrastructures of fact-checking](https://mlsanity.com/paper/2205.10894/) (2022)
- [Community-Based Fact-Checking on Twitter's Birdwatch Platform](https://mlsanity.com/paper/2104.07175/) (2021)
- [Did the Roll-Out of Community Notes Reduce Engagement With Misinformation on X/Twitter?](https://mlsanity.com/paper/2307.07960/) (2023)
- [Beyond the Crowd: LLM-Augmented Community Notes for Governing Health Misinformation](https://mlsanity.com/paper/2510.11423/) (2025)
- [SciClops: Detecting and Contextualizing Scientific Claims for Assisting Manual Fact-Checking](https://mlsanity.com/paper/2110.13090/) (2021)
- [Can Community Notes Replace Professional Fact-Checkers?](https://mlsanity.com/paper/2502.14132/) (2025)
- [Learning Human-like Representations to Enable Learning Human Values](https://mlsanity.com/paper/2312.14106/) (2023)
- [COMMUNITYNOTES: A Dataset for Exploring the Helpfulness of Fact-Checking Explanations](https://mlsanity.com/paper/2510.24810/) (2025)
- [Consensus Stability of Community Notes on X](https://mlsanity.com/paper/2601.14002/) (2026)
- [Collab: Fostering Critical Identification of Deepfake Videos on Social Media via Synergistic Annotation](https://mlsanity.com/paper/2601.17371/) (2026)
- [Fact-Checking Comments Precede Community Notes to Misleading Posts on X](https://mlsanity.com/paper/2509.11052/) (2025)
- [Request a Note: How the Request Function Shapes X's Community Notes System](https://mlsanity.com/paper/2509.09956/) (2025)

## Abstract (arXiv)

Community-based fact-checking that relies on cross-consensus is expanding rapidly on social media platforms. However, the delay and low-ratio of cross-consensus community fact-checks rated by human contributors remains a significant challenge. To address this, we first created ComRate, a large-scale dataset comprising 2.5 million community notes and over 209 million ratings sourced from mathbbX. We then propose MultiCom, a persona-guided multi-agent rating framework for community note evaluation. MultiCom simulates diverse rater population by clustering contributors in a matrix-factorized rater space and prompting persona agents to generate structured assessments based on the official community notes rating schema. These agents output structured and explainable judgments, such as confidence, agreement signals and reasons. An out-of-fold calibrated aggregation algorithm combines features such as raw votes and diagnostic reason signals for reliable prediction. Extensive evaluations demonstrate that MultiCom outperforms alternative methods, achieving an average accuracy of 84.7% (balanced accuracy 68.3%, macro-F1 60.1%) on the evaluation set.