mlsanityData to 7 Oct 2026

CompassJudger-1: All-in-one Judge Model Helps Model Evaluation and Evolution

arXiv 2410.16256 ↗First posted 21 Oct 2024cs.CL cs.AIcited by 42

Not on the map. Why?

Abstract (arXiv)

Efficient and accurate evaluation is crucial for the continuous improvement of large language models (LLMs). Among various assessment methods, subjective evaluation has garnered significant attention due to its superior alignment with real-world usage scenarios and human preferences. However, human-based evaluations are costly and lack reproducibility, making precise automated evaluators (judgers) vital in this process. In this report, we introduce CompassJudger-1, the first open-source all-in-one judge LLM. CompassJudger-1 is a general-purpose LLM that demonstrates remarkable versatility. It is capable of: 1. Performing unitary scoring and two-model comparisons as a reward model; 2. Conducting evaluations according to specified formats; 3. Generating critiques; 4. Executing diverse tasks like a general LLM. To assess the evaluation capabilities of different judge models under a unified setting, we have also established JudgerBench, a new benchmark that encompasses various subjective evaluation tasks and covers a wide range of topics. CompassJudger-1 offers a comprehensive solution for various evaluation tasks while maintaining the flexibility to adapt to diverse requirements. Both CompassJudger and JudgerBench are released and available to the research community athttps://github.com/open-compass/CompassJudger. We believe that by open-sourcing these tools, we can foster collaboration and accelerate progress in LLM evaluation methodologies.

Key facts. Cited by 42 papers in the map (data through 7 Oct 2026). Builds on 15 of its 22 references that are in the map. First posted on arXiv 21 Oct 2024.

Builds on

15 of 22 references.

References · 15

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2024Qwen2 Technical Reportcited by 2,076
  3. 2024Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preferencecited by 1,151
  4. 2024Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorscited by 865
  5. 2023AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedbackcited by 738
  6. 2024From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipelinecited by 484
  7. 2024RewardBench: Evaluating Reward Models for Language Modelingcited by 480
  8. 2024InternLM2 Technical Reportcited by 391
  9. 2023JudgeLM: Fine-tuned Large Language Models are Scalable Judgescited by 351
  10. 2023PandaLM: An Automatic Evaluation Benchmark for LLM Instruction Tuning Optimizationcited by 311
  11. 2024WildBench: Benchmarking LLMs with Challenging Tasks from Real Users in the Wildcited by 211
  12. 2023Generative Judge for Evaluating Alignmentcited by 163
3 more
  1. 2024The BiGGen Bench: A Principled Benchmark for Fine-grained Evaluation of Language Models with Language Modelscited by 89
  2. 2024FOFO: A Benchmark to Evaluate LLMs' Format-Following Capabilitycited by 79
  3. 2023AlignBench: Benchmarking Chinese Alignment of Large Language Modelscited by 68

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 30; 2026: 11.

42 papers.

Sort:

By area:

  1. 2024LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation Methodscited by 451
  2. 2025Inference-Time Scaling for Generalist Reward Modelingcited by 233
  3. 2025J1: Incentivizing Thinking in LLM-as-a-Judge via Reinforcement Learningcited by 73
  4. 2025xVerify: Efficient Answer Verifier for Reasoning Model Evaluationscited by 53
  5. 2025A Comprehensive Survey of Reward Models: Taxonomy, Applications, Challenges, and Futurecited by 53
  6. 2025Agentic Reward Modeling: Integrating Human Preferences with Verifiable Correctness Signals for Reliable Reward Systemscited by 49
  7. 2025Reward Reasoning Modelcited by 47
  8. 2025Writing-Zero: Bridge the Gap Between Non-verifiable Tasks and Verifiable Rewardscited by 47
  9. 2025CompassVerifier: A Unified and Robust Verifier for LLMs Evaluation and Outcome Rewardcited by 39
  10. 2025Improving LLM-as-a-Judge Inference with the Judgment Distributioncited by 36
  11. 2025Improve LLM-as-a-Judge Ability as a General Abilitycited by 35
  12. 2025Genius: A Generalizable and Purely Unsupervised Self-Training Framework For Advanced Reasoningcited by 24
30 more
  1. 2025Think-J: Learning to Think for Generative LLM-as-a-Judgecited by 20
  2. 2025Condor: Enhance LLM Alignment with Knowledge-Driven Data Synthesis and Refinementcited by 17
  3. 2025When AIs Judge AIs: The Rise of Agent-as-a-Judge Evaluation for LLMscited by 16
  4. 2025Sailing by the Stars: A Survey on Reward Models and Learning Strategies for Learning from Rewardscited by 14
  5. 2025MIG: Automatic Data Selection for Instruction Tuning by Maximizing Information Gain in Semantic Spacecited by 12
  6. 2025Scaling Physical Reasoning with the PHYSICS Datasetcited by 12
  7. 2025CompassJudger-2: Towards Generalist Judge Model via Verifiable Rewardscited by 11
  8. 2025J4R: Learning to Judge with Equivalent Initial State Group Relative Policy Optimizationcited by 9
  9. 2025Training Language Model to Critique for Better Refinementcited by 7
  10. 2025Enhancing Large Language Model Reasoning with Reward Models: An Analytical Surveycited by 7
  11. 2025RedOne: Revealing Domain-specific LLM Post-Training in Social Networking Servicescited by 6
  12. 2026Reason Only When Needed: Efficient Generative Reward Modeling via Model-Internal Uncertaintycited by 6
  13. 2025Foundational Automatic Evaluators: Scaling Multi-Task Generative Evaluator Training for Reasoning-Centric Domainscited by 5
  14. 2026CompliBench: Benchmarking LLM Judges for Compliance Violation Detection in Dialogue Systemscited by 5
  15. 2025Cheems: A Practical Guidance for Building and Evaluating Chinese Reward Models from Scratchcited by 2
  16. 2025Act-Adaptive Margin: Dynamically Calibrating Reward Models for Subjective Ambiguitycited by 2
  17. 2025Evaluating Text Style Transfer: A Nine-Language Benchmark for Text Detoxificationcited by 2
  18. 2026Rewarding Creativity: A Human-Aligned Generative Reward Model for Reinforcement Learning in Storytellingcited by 2
  19. 2026Balanced Aggregation: Understanding and Fixing Aggregation Bias in GRPOcited by 2
  20. 2026FormalRewardBench: A Benchmark for Formal Theorem Proving Reward Modelscited by 2
  21. 2025OphthBench: A Comprehensive Benchmark for Evaluating Large Language Models in Chinese Ophthalmologycited by 1
  22. 2025A Dual-Perspective NLG Meta-Evaluation Framework with Automatic Benchmark and Better Interpretabilitycited by 1
  23. 2025SCAN: Structured Capability Assessment and Navigation for LLMscited by 1
  24. 2026IRPM: Intergroup Relative Preference Modeling for Pointwise Generative Reward Modelscited by 1
  25. 2026CE-RM: A Pointwise Generative Reward Model Optimized via Two-Stage Rollout and Unified Criteriacited by 1
  26. 2025Routing to the Right Expertise: A Trustworthy Judge for Instruction-based Image Editingcited by 0
  27. 2026ReLE: A Scalable System and Structured Benchmark for Diagnosing Capability Anisotropy in Chinese LLMscited by 0
  28. 2026MeetBench-XL: Calibrated Multi-Dimensional Evaluation and Learned Dual-Policy Agents for Real-Time Meetingscited by 0
  29. 2026TrustRoboReward: Preference-Ordered Isotonic Score Editing for Multi-Paradigm Robot Reward Modelscited by 0
  30. 2026OmniJudge or OmniBias? Diagnosing Multimodal Judges through Balanced, Decoupled Lensescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026JudgeArena: A Unified Framework for Reproducible LLM-Judge Evaluation
  2. 2025Leveraging LLMs as Meta-Judges: A Multi-Agent Framework for Evaluating LLM Judgments
  3. 2026BabelJudge: Measuring LLM-as-a-Judge Reliability Across Languages and Agent Trajectories
  4. 2024Judging the Judges: Evaluating Alignment and Vulnerabilities in LLMs-as-Judges
  5. 2026When Consistency Does Not Mean Reliability: Evaluating Local LLM Judges Against Human Ratings
  6. 2024Can Large Language Models be Trusted for Evaluation? Scalable Meta-Evaluation of LLMs as Evaluators via Agent Debate
  7. 2024MM-Eval: A Multilingual Meta-Evaluation Benchmark for LLM-as-a-Judge and Reward Models
  8. 2025CodeJudgeBench: Benchmarking LLM-as-a-Judge for Coding Tasks