# InfiCoEvalChain: A Blockchain-Based Decentralized Framework for Collaborative LLM Evaluation

arXiv: https://arxiv.org/abs/2602.08229 · HTML: https://mlsanity.com/paper/2602.08229/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 35 references that are in the map.
- First posted on arXiv 9 Feb 2026.

## Builds on (most cited first, 27 of 27)

- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [TruthfulQA: Measuring How Models Mimic Human Falsehoods](https://mlsanity.com/paper/2109.07958/) (2021)
- [LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code](https://mlsanity.com/paper/2403.07974/) (2024)
- [Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference](https://mlsanity.com/paper/2403.04132/) (2024)
- [BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Dataset](https://mlsanity.com/paper/2307.04657/) (2023)
- [BBQ: A Hand-Built Bias Benchmark for Question Answering](https://mlsanity.com/paper/2110.08193/) (2021)
- [Adding Error Bars to Evals: A Statistical Approach to Language Model Evaluations](https://mlsanity.com/paper/2411.00640/) (2024)
- [Benchmarking Distributional Alignment of Large Language Models](https://mlsanity.com/paper/2411.05403/) (2024)
- [SG-Bench: Evaluating LLM Safety Generalization Across Diverse Tasks and Prompt Types](https://mlsanity.com/paper/2410.21965/) (2024)
- [R-Bench: Graduate-level Multi-disciplinary Benchmarks for LLM & MLLM Complex Reasoning Evaluation](https://mlsanity.com/paper/2505.02018/) (2025)
- [mABC: multi-Agent Blockchain-Inspired Collaboration for root cause analysis in micro-services architecture](https://mlsanity.com/paper/2404.12135/) (2024)
- [Benchmarking LLMs' Judgments with No Gold Standard](https://mlsanity.com/paper/2411.07127/) (2024)
- [Sample-Efficient Human Evaluation of Large Language Models via Maximum Discrepancy Competition](https://mlsanity.com/paper/2404.08008/) (2024)
- [Bias and Volatility: A Statistical Framework for Evaluating Large Language Model's Stereotypes and the Associated Generation Inconsistency](https://mlsanity.com/paper/2402.15481/) (2024)
- [Benchmark^2: Systematic Evaluation of LLM Benchmarks](https://mlsanity.com/paper/2601.03986/) (2026)
- [InfiGFusion: Graph-on-Logits Distillation via Efficient Gromov-Wasserstein for Model Fusion](https://mlsanity.com/paper/2505.13893/) (2025)
- [Model Merging Scaling Laws in Large Language Models](https://mlsanity.com/paper/2509.24244/) (2025)
- [SafeWorld: Geo-Diverse Safety Alignment](https://mlsanity.com/paper/2412.06483/) (2024)
- [Live Graph Lab: Towards Open, Dynamic and Real Transaction Graphs with NFT](https://mlsanity.com/paper/2310.11709/) (2023)
- [SCALM: Detecting Bad Practices in Smart Contracts Through LLMs](https://mlsanity.com/paper/2502.04347/) (2025)
- [FLEX: A Benchmark for Evaluating Robustness of Fairness in Large Language Models](https://mlsanity.com/paper/2503.19540/) (2025)
- [LLM-Net: Democratizing LLMs-as-a-Service through Blockchain-based Expert Networks](https://mlsanity.com/paper/2501.07288/) (2025)
- [AIArena: A Blockchain-Based Decentralized AI Training Platform](https://mlsanity.com/paper/2412.14566/) (2024)
- [Know Your Intent: An Autonomous Multi-Perspective LLM Agent Framework for DeFi User Transaction Intent Mining](https://mlsanity.com/paper/2511.15456/) (2025)
- [Blockchain Meets LLMs: A Living Survey on Bidirectional Integration](https://mlsanity.com/paper/2411.16809/) (2024)

## Built on by (most cited first, 3 of 3)

- [Geometry Conflict: Explaining and Controlling Forgetting in LLM Continual Post-Training](https://mlsanity.com/paper/2605.09608/) (2026)
- [FeatCal: Feature Calibration for Post-Merging Models](https://mlsanity.com/paper/2605.13030/) (2026)
- [Who Verifies the Benchmark? Decentralizing Trust in Large Language Model Evaluation](https://mlsanity.com/paper/2608.07762/) (2026)

## Abstract (arXiv)

The rapid advancement of large language models (LLMs) demands increasingly reliable evaluation, yet current centralized evaluation suffers from opacity, overfitting, and hardware-induced variance. Our empirical analysis reveals an alarming inconsistency in existing evaluations: the standard deviation across ten repeated runs of a single model on HumanEval (1.67) actually exceeds the performance gap among the top-10 models on the official leaderboard (0.91), rendering current rankings statistically precarious. To mitigate these instabilities, we propose a decentralized evaluation framework that enables hardware and parameter diversity through large-scale benchmarking across heterogeneous compute nodes. By leveraging the blockchain-based protocol, the framework incentivizes global contributors to act as independent validators, using a robust reward system to ensure evaluation integrity and discourage dishonest participation. This collective verification transforms evaluation from a "centralized black box" into a "decentralized endorsement" where multi-party consensus and diverse inference environments yield a more stable, representative metric. Experimental results demonstrate that the decentralized evaluation framework reduces the standard deviation across ten runs on the same model to 0.28. This significant improvement over conventional frameworks ensures higher statistical confidence in model rankings. We have completely implemented this platform and will soon release it to the community.