# CS-Bench: A Comprehensive Benchmark for Large Language Models towards Computer Science Mastery

arXiv: https://arxiv.org/abs/2406.08587 · HTML: https://mlsanity.com/paper/2406.08587/

## Key facts

- Cited by 30 papers in the map (data through 14 Oct 2026).
- Builds on 44 of its 76 references that are in the map.
- First posted on arXiv 12 Jun 2024.

## Builds on (most cited first, 30 of 44)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://mlsanity.com/paper/2309.06180/) (2023)
- [Measuring Mathematical Problem Solving With the MATH Dataset](https://mlsanity.com/paper/2103.03874/) (2021)
- [Program Synthesis with Large Language Models](https://mlsanity.com/paper/2108.07732/) (2021)
- [Qwen Technical Report](https://mlsanity.com/paper/2309.16609/) (2023)
- [Mistral 7B](https://mlsanity.com/paper/2310.06825/) (2023)
- [GPQA: A Graduate-Level Google-Proof Q&A Benchmark](https://mlsanity.com/paper/2311.12022/) (2023)
- [Mixtral of Experts](https://mlsanity.com/paper/2401.04088/) (2024)
- [Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them](https://mlsanity.com/paper/2210.09261/) (2022)
- [HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusal](https://mlsanity.com/paper/2402.04249/) (2024)
- [A Survey on Evaluation of Large Language Models](https://mlsanity.com/paper/2307.03109/) (2023)
- [Instruction-Following Evaluation for Large Language Models](https://mlsanity.com/paper/2311.07911/) (2023)
- [PaLM 2 Technical Report](https://mlsanity.com/paper/2305.10403/) (2023)
- [GLM-130B: An Open Bilingual Pre-trained Model](https://mlsanity.com/paper/2210.02414/) (2022)
- [Large Language Models](https://mlsanity.com/paper/2307.05782/) (2023)
- [Baichuan 2: Open Large-scale Language Models](https://mlsanity.com/paper/2309.10305/) (2023)
- [DeepSeek LLM: Scaling Open-Source Language Models with Longtermism](https://mlsanity.com/paper/2401.02954/) (2024)
- [MAmmoTH: Building Math Generalist Models through Hybrid Instruction Tuning](https://mlsanity.com/paper/2309.05653/) (2023)
- [Scaling Relationship on Learning Mathematical Reasoning with Large Language Models](https://mlsanity.com/paper/2308.01825/) (2023)
- [InternLM2 Technical Report](https://mlsanity.com/paper/2403.17297/) (2024)
- [LegalBench: A Collaboratively Built Benchmark for Measuring Legal Reasoning in Large Language Models](https://mlsanity.com/paper/2308.11462/) (2023)
- [Reasoning or Reciting? Exploring the Capabilities and Limitations of Language Models Through Counterfactual Tasks](https://mlsanity.com/paper/2307.02477/) (2023)
- [L-Eval: Instituting Standardized Evaluation for Long Context Language Models](https://mlsanity.com/paper/2307.11088/) (2023)
- [How Abilities in Large Language Models are Affected by Supervised Fine-tuning Data Composition](https://mlsanity.com/paper/2310.05492/) (2023)
- [Evaluating the Logical Reasoning Ability of ChatGPT and GPT-4](https://mlsanity.com/paper/2304.03439/) (2023)

## Built on by (most cited first, 30 of 30)

- [Agent Laboratory: Using LLM Agents as Research Assistants](https://mlsanity.com/paper/2501.04227/) (2025)
- [We-Math: Does Your Large Multimodal Model Achieve Human-like Mathematical Reasoning?](https://mlsanity.com/paper/2407.01284/) (2024)
- [AgentRxiv: Towards Collaborative Autonomous Research](https://mlsanity.com/paper/2503.18102/) (2025)
- [MegaScience: Pushing the Frontiers of Post-Training Datasets for Science Reasoning](https://mlsanity.com/paper/2507.16812/) (2025)
- [We-Math 2.0: A Versatile MathBook System for Incentivizing Visual Mathematical Reasoning](https://mlsanity.com/paper/2508.10433/) (2025)
- [SimRAG: Self-Improving Retrieval-Augmented Generation for Adapting Large Language Models to Specialized Domains](https://mlsanity.com/paper/2410.17952/) (2024)
- [OJBench: A Competition Level Code Benchmark For Large Language Models](https://mlsanity.com/paper/2506.16395/) (2025)
- [V-Thinker: Interactive Thinking with Images](https://mlsanity.com/paper/2511.04460/) (2025)
- [Progressive Multimodal Reasoning via Active Retrieval](https://mlsanity.com/paper/2412.14835/) (2024)
- [Reward and Guidance through Rubrics: Promoting Exploration to Improve Multi-Domain Reasoning](https://mlsanity.com/paper/2511.12344/) (2025)
- [FrontendBench: A Benchmark for Evaluating LLMs on Front-End Development via Automatic Evaluation](https://mlsanity.com/paper/2506.13832/) (2025)
- [DotaMath: Decomposition of Thought with Code Assistance and Self-correction for Mathematical Reasoning](https://mlsanity.com/paper/2407.04078/) (2024)
- [Human Decision-making is Susceptible to AI-driven Manipulation](https://mlsanity.com/paper/2502.07663/) (2025)
- [MedReasoner: Reinforcement Learning Drives Reasoning Grounding from Clinical Thought to Pixel-Level Precision](https://mlsanity.com/paper/2508.08177/) (2025)
- [SEAS: Self-Evolving Adversarial Safety Optimization for Large Language Models](https://mlsanity.com/paper/2408.02632/) (2024)
- [How Do Your Code LLMs Perform? Empowering Code Instruction Tuning with High-Quality Data](https://mlsanity.com/paper/2409.03810/) (2024)
- [TREX: Automating LLM Fine-tuning via Agent-Driven Tree-based Exploration](https://mlsanity.com/paper/2604.14116/) (2026)
- [XtraGPT: Context-Aware and Controllable Academic Paper Revision via Human-AI Collaboration](https://mlsanity.com/paper/2505.11336/) (2025)
- [Beyond Single-Value Metrics: Evaluating and Enhancing LLM Unlearning with Cognitive Diagnosis](https://mlsanity.com/paper/2502.13996/) (2025)
- [BenchBench: Benchmarking Automated Benchmark Generation](https://mlsanity.com/paper/2603.20807/) (2026)
- [KRISTEVA: Close Reading as a Novel Task for Benchmarking Interpretive Reasoning](https://mlsanity.com/paper/2505.09825/) (2025)
- [AI Research Agents Narrow Scientific Exploration](https://mlsanity.com/paper/2605.27905/) (2026)
- [GuessArena: Guess Who I Am? A Self-Adaptive Framework for Evaluating LLMs in Domain-Specific Knowledge and Reasoning](https://mlsanity.com/paper/2505.22661/) (2025)
- [RAG in the Wild: On the (In)effectiveness of LLMs with Mixture-of-Knowledge Retrieval Augmentation](https://mlsanity.com/paper/2507.20059/) (2025)
- [Evaluating Multimodal Generative AI with Korean Educational Standards](https://mlsanity.com/paper/2502.15422/) (2025)
- [Multi-Dimensional Insights: Benchmarking Real-World Personalization in Large Multimodal Models](https://mlsanity.com/paper/2412.12606/) (2024)
- [Challenges for AI in Multimodal STEM Assessments: a Human-AI Comparison](https://mlsanity.com/paper/2507.03013/) (2025)
- [KARMA: Knowledge graph-based Automated Reasoning Materialization and Alignment](https://mlsanity.com/paper/2607.03166/) (2026)
- [SPARK: Skeleton-Guided Reasoning Synthesis from Large-Scale Scientific Literature](https://mlsanity.com/paper/2608.30214/) (2026)
- [S3-Bench: Evaluating Speech Interaction Models as Scientific Voice Assistants](https://mlsanity.com/paper/2609.09852/) (2026)

## Abstract (arXiv)

Large language models (LLMs) have demonstrated significant potential in advancing various fields of research and society. However, the current community of LLMs overly focuses on benchmarks for analyzing specific foundational skills (e.g. mathematics and code generation), neglecting an all-round evaluation of the computer science field. To bridge this gap, we introduce CS-Bench, the first multilingual (English, Chinese, French, German) benchmark dedicated to evaluating the performance of LLMs in computer science. CS-Bench comprises approximately 10K meticulously curated test samples, covering 26 subfields across 4 key areas of computer science, encompassing various task forms and divisions of knowledge and reasoning. Utilizing CS-Bench, we conduct a comprehensive evaluation of over 30 mainstream LLMs, revealing the relationship between CS performance and model scales. We also quantitatively analyze the reasons for failures in existing LLMs and highlight directions for improvements, including knowledge supplementation and CS-specific reasoning. Further cross-capability experiments show a high correlation between LLMs' capabilities in computer science and their abilities in mathematics and coding. Moreover, expert LLMs specialized in mathematics and coding also demonstrate strong performances in several CS subfields. Looking ahead, we envision CS-Bench serving as a cornerstone for LLM applications in the CS field and paving new avenues in assessing LLMs' diverse reasoning capabilities. The CS-Bench data and evaluation code are available at https://github.com/csbench/csbench.