mlsanityData to 7 Oct 2026

CS-Bench: A Comprehensive Benchmark for Large Language Models towards Computer Science Mastery

arXiv 2406.08587 ↗First posted 12 Jun 2024cs.CL cs.AI cs.LGcited by 30

Not on the map. Why?

Abstract (arXiv)

Large language models (LLMs) have demonstrated significant potential in advancing various fields of research and society. However, the current community of LLMs overly focuses on benchmarks for analyzing specific foundational skills (e.g. mathematics and code generation), neglecting an all-round evaluation of the computer science field. To bridge this gap, we introduce CS-Bench, the first multilingual (English, Chinese, French, German) benchmark dedicated to evaluating the performance of LLMs in computer science. CS-Bench comprises approximately 10K meticulously curated test samples, covering 26 subfields across 4 key areas of computer science, encompassing various task forms and divisions of knowledge and reasoning. Utilizing CS-Bench, we conduct a comprehensive evaluation of over 30 mainstream LLMs, revealing the relationship between CS performance and model scales. We also quantitatively analyze the reasons for failures in existing LLMs and highlight directions for improvements, including knowledge supplementation and CS-specific reasoning. Further cross-capability experiments show a high correlation between LLMs' capabilities in computer science and their abilities in mathematics and coding. Moreover, expert LLMs specialized in mathematics and coding also demonstrate strong performances in several CS subfields. Looking ahead, we envision CS-Bench serving as a cornerstone for LLM applications in the CS field and paving new avenues in assessing LLMs' diverse reasoning capabilities. The CS-Bench data and evaluation code are available at https://github.com/csbench/csbench.

Key facts. Cited by 30 papers in the map (data through 7 Oct 2026). Builds on 44 of its 76 references that are in the map. First posted on arXiv 12 Jun 2024.

Builds on

44 of 76 references.

References · 44

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  4. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  5. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  6. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  7. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  8. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  9. 2021Program Synthesis with Large Language Modelscited by 3,509
  10. 2023Qwen Technical Reportcited by 3,273
  11. 2023Mistral 7Bcited by 3,065
  12. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,748
32 more
  1. 2024Mixtral of Expertscited by 1,785
  2. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,767
  3. 2024HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusalcited by 1,330
  4. 2023A Survey on Evaluation of Large Language Modelscited by 1,227
  5. 2023PaLM 2 Technical Reportcited by 1,154
  6. 2023Instruction-Following Evaluation for Large Language Modelscited by 1,143
  7. 2022GLM-130B: An Open Bilingual Pre-trained Modelcited by 837
  8. 2023Large Language Modelscited by 774
  9. 2023Baichuan 2: Open Large-scale Language Modelscited by 664
  10. 2024DeepSeek LLM: Scaling Open-Source Language Models with Longtermismcited by 621
  11. 2023MAmmoTH: Building Math Generalist Models through Hybrid Instruction Tuningcited by 483
  12. 2023Scaling Relationship on Learning Mathematical Reasoning with Large Language Modelscited by 417
  13. 2024InternLM2 Technical Reportcited by 391
  14. 2023LegalBench: A Collaboratively Built Benchmark for Measuring Legal Reasoning in Large Language Modelscited by 328
  15. 2023Reasoning or Reciting? Exploring the Capabilities and Limitations of Language Models Through Counterfactual Taskscited by 243
  16. 2023L-Eval: Instituting Standardized Evaluation for Long Context Language Modelscited by 232
  17. 2023How Abilities in Large Language Models are Affected by Supervised Fine-tuning Data Compositioncited by 222
  18. 2023Evaluating the Logical Reasoning Ability of ChatGPT and GPT-4cited by 202
  19. 2023What can Large Language Models do in chemistry? A comprehensive benchmark on eight taskscited by 179
  20. 2023A Survey of Large Language Models in Medicine: Progress, Application, and Challengecited by 123
  21. 2023T-Eval: Evaluating the Tool Utilization Capability of Large Language Models Step by Stepcited by 91
  22. 2024Large Language Models for Robotics: Opportunities, Challenges, and Perspectivescited by 88
  23. 2024Revolutionizing Finance with LLMs: An Overview of Applications and Insightscited by 88
  24. 2023Testing the General Deductive Reasoning Capacity of Large Language Models Using OOD Examplescited by 84
  25. 2023Unifying the Perspectives of NLP and Software Engineering: A Survey on Language Models for Codecited by 62
  26. 2023Robot Learning in the Era of Foundation Models: A Surveycited by 37
  27. 2023M3KE: A Massive Multi-Level Multi-Subject Knowledge Evaluation Benchmark for Chinese Large Language Modelscited by 30
  28. 2023Applications of Large Scale Foundation Models for Autonomous Drivingcited by 14
  29. 2024Beyond the Known: Investigating LLMs Performance on Out-of-Domain Intent Detectioncited by 14
  30. 2024DolphCoder: Echo-Locating Code Large Language Models with Diverse and Multi-Objective Instruction Tuningcited by 11
  31. 2023Large Language Models Illuminate a Progressive Pathway to Artificial Healthcare Assistant: A Reviewcited by 8
  32. 2024Sentinels of the Stream: Unleashing Large Language Models for Dynamic Packet Classification in Software Defined Networks -- Position Papercited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 7; 2025: 17; 2026: 6.

30 papers.

Sort:

By area:

  1. 2025Agent Laboratory: Using LLM Agents as Research Assistantscited by 405
  2. 2024We-Math: Does Your Large Multimodal Model Achieve Human-like Mathematical Reasoning?cited by 281
  3. 2025AgentRxiv: Towards Collaborative Autonomous Researchcited by 64
  4. 2025MegaScience: Pushing the Frontiers of Post-Training Datasets for Science Reasoningcited by 56
  5. 2025We-Math 2.0: A Versatile MathBook System for Incentivizing Visual Mathematical Reasoningcited by 32
  6. 2024SimRAG: Self-Improving Retrieval-Augmented Generation for Adapting Large Language Models to Specialized Domainscited by 28
  7. 2025V-Thinker: Interactive Thinking with Imagescited by 26
  8. 2024Progressive Multimodal Reasoning via Active Retrievalcited by 24
  9. 2025OJBench: A Competition Level Code Benchmark For Large Language Modelscited by 22
  10. 2024DotaMath: Decomposition of Thought with Code Assistance and Self-correction for Mathematical Reasoningcited by 19
  11. 2025Human Decision-making is Susceptible to AI-driven Manipulationcited by 19
  12. 2025Reward and Guidance through Rubrics: Promoting Exploration to Improve Multi-Domain Reasoningcited by 18
18 more
  1. 2025FrontendBench: A Benchmark for Evaluating LLMs on Front-End Development via Automatic Evaluationcited by 17
  2. 2025MedReasoner: Reinforcement Learning Drives Reasoning Grounding from Clinical Thought to Pixel-Level Precisioncited by 10
  3. 2024How Do Your Code LLMs Perform? Empowering Code Instruction Tuning with High-Quality Datacited by 8
  4. 2024SEAS: Self-Evolving Adversarial Safety Optimization for Large Language Modelscited by 7
  5. 2025XtraGPT: Context-Aware and Controllable Academic Paper Revision via Human-AI Collaborationcited by 6
  6. 2026TREX: Automating LLM Fine-tuning via Agent-Driven Tree-based Explorationcited by 6
  7. 2025Beyond Single-Value Metrics: Evaluating and Enhancing LLM Unlearning with Cognitive Diagnosiscited by 5
  8. 2025KRISTEVA: Close Reading as a Novel Task for Benchmarking Interpretive Reasoningcited by 4
  9. 2025GuessArena: Guess Who I Am? A Self-Adaptive Framework for Evaluating LLMs in Domain-Specific Knowledge and Reasoningcited by 3
  10. 2025RAG in the Wild: On the (In)effectiveness of LLMs with Mixture-of-Knowledge Retrieval Augmentationcited by 3
  11. 2026BenchBench: Benchmarking Automated Benchmark Generationcited by 3
  12. 2025Evaluating Multimodal Generative AI with Korean Educational Standardscited by 2
  13. 2026AI Research Agents Narrow Scientific Explorationcited by 2
  14. 2024Multi-Dimensional Insights: Benchmarking Real-World Personalization in Large Multimodal Modelscited by 1
  15. 2025Challenges for AI in Multimodal STEM Assessments: a Human-AI Comparisoncited by 1
  16. 2026KARMA: Knowledge graph-based Automated Reasoning Materialization and Alignmentcited by 0
  17. 2026SPARK: Skeleton-Guided Reasoning Synthesis from Large-Scale Scientific Literaturecited by 0
  18. 2026S3-Bench: Evaluating Speech Interaction Models as Scientific Voice Assistantscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024CLR-Bench: Evaluating Large Language Models in College-level Reasoning
  2. 2025Cross-Task Benchmarking and Evaluation of General-Purpose and Code-Specific Large Language Models ↗
  3. 2025LLM-ProS: Analyzing Large Language Models' Performance in Competitive Problem Solving
  4. 2023CodeApex: A Bilingual Programming Evaluation Benchmark for Large Language Models
  5. 2026CodeGolf Bench: A Multi-Language Benchmark for Evaluating Concise Code Generation Capabilities of Large Language Models
  6. 2026EsoLang-Bench: Evaluating Genuine Reasoning in Large Language Models via Esoteric Programming Languages
  7. 2024FineMath: A Fine-Grained Mathematical Evaluation Benchmark for Chinese Large Language Models
  8. 2024LLMs Are Not Intelligent Thinkers: Introducing Mathematical Topic Tree Benchmark for Comprehensive Evaluation of LLMs