mlsanityData to 7 Oct 2026

BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructions

arXiv 2406.15877 ↗First posted 22 Jun 2024cs.SE cs.AI cs.CLcited by 527

Not on the map. Why?

Cited by 124 in LLM agents and tool use, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Task automation has been greatly empowered by the recent advances in Large Language Models (LLMs) via Python code, where the tasks ranging from software engineering development to general-purpose reasoning. While current benchmarks have shown that LLMs can solve tasks using programs like human developers, the majority of their evaluations are limited to short and self-contained algorithmic tasks or standalone function calls. Solving challenging and practical tasks requires the capability of utilizing diverse function calls as tools to efficiently implement functionalities like data analysis and web development. In addition, using multiple tools to solve a task needs compositional reasoning by accurately understanding complex instructions. Fulfilling both of these characteristics can pose a great challenge for LLMs.To assess how well LLMs can solve challenging and practical tasks via programs, we introduce BigCodeBench, a benchmark that challenges LLMs to invoke multiple function calls as tools from 139 libraries and 7 domains for 1,140 fine-grained tasks. To evaluate LLMs rigorously, each task encompasses 5.6 test cases with an average branch coverage of 99%. In addition, we propose a natural-language-oriented variant of BigCodeBench, BigCodeBench-Instruct, that automatically transforms the original docstrings into short instructions only with essential information. Our extensive evaluation of 60 LLMs shows that LLMs are not yet capable of following complex instructions to use function calls precisely, with scores up to 60%, significantly lower than the human performance of 97%. The results underscore the need for further advancements in this area.

Key facts. Cited by 527 papers in the map (data through 7 Oct 2026). Builds on 85 of its 140 references that are in the map. First posted on arXiv 22 Jun 2024.

Builds on

85 of 140 references.

References · 85

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  4. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  5. 2020Scaling Laws for Neural Language Modelscited by 6,411
  6. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  7. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,064
  8. 2021Program Synthesis with Large Language Modelscited by 3,509
  9. 2023Qwen Technical Reportcited by 3,273
  10. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,193
  11. 2023Code Llama: Open Foundation Models for Codecited by 2,105
  12. 2023WebArena: A Realistic Web Environment for Building Autonomous Agentscited by 1,908
38 more
  1. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 1,906
  2. 2024SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineeringcited by 1,628
  3. 2023The Rise and Potential of Large Language Model Based Agents: A Surveycited by 1,422
  4. 2018Datasheets for Datasetscited by 1,371
  5. 2023Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generationcited by 1,332
  6. 2023Gorilla: Large Language Model Connected with Massive APIscited by 1,302
  7. 2024DeepSeek-Coder: When the Large Language Model Meets Programming -- The Rise of Code Intelligencecited by 1,154
  8. 2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Modelcited by 1,093
  9. 2023Teaching Large Language Models to Self-Debugcited by 963
  10. 2021Measuring Coding Challenge Competence With APPScited by 879
  11. 2022CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesiscited by 876
  12. 2023WizardCoder: Empowering Code Large Language Models with Evol-Instructcited by 728
  13. 2024Yi: Open Foundation Models by 01.AIcited by 706
  14. 2024LLM Evaluators Recognize and Favor Their Own Generationscited by 651
  15. 2022PAL: Program-aided Language Modelscited by 635
  16. 2022Contrastive Decoding: Open-ended Text Generation as Optimizationcited by 598
  17. 2022InCoder: A Generative Model for Code Infilling and Synthesiscited by 492
  18. 2024StarCoder 2 and The Stack v2: The Next Generationcited by 484
  19. 2022DS-1000: A Natural and Reliable Benchmark for Data Science Code Generationcited by 442
  20. 2023Large Language Models for Software Engineering: A Systematic Literature Reviewcited by 389
  21. 2023SOTOPIA: Interactive Evaluation for Social Intelligence in Language Agentscited by 350
  22. 2018Speaker-Follower Models for Vision-and-Language Navigationcited by 337
  23. 2023Tool Learning with Foundation Modelscited by 302
  24. 2023Generating Images with Multimodal Language Modelscited by 259
  25. 2022Neural Theory-of-Mind? On the Limits of Social Intelligence in Large LMscited by 220
  26. 2023Rethinking Benchmark and Contamination for Language Models with Rephrased Samplescited by 192
  27. 2023CrossCodeEval: A Diverse and Multilingual Benchmark for Cross-File Code Completioncited by 181
  28. 2023OctoPack: Instruction Tuning Code Large Language Modelscited by 178
  29. 2023Large Language Models for Software Engineering: Survey and Open Problemscited by 178
  30. 2023SantaCoder: don't reach for the stars!cited by 169
  31. 2023Software Testing with Large Language Models: Survey, Landscape, and Visioncited by 163
  32. 2022Large Language Models Meet NL2Code: A Surveycited by 135
  33. 2023Grounding Language Models to Images for Multimodal Inputs and Outputscited by 129
  34. 2023Keep the Conversation Going: Fixing 162 out of 337 bugs for 0.42 each using ChatGPTcited by 125
  35. 2022Natural Language to Code Translation with Executioncited by 122
  36. 2024Dissecting Adversarial Robustness of Multimodal LM Agentscited by 118
  37. 2024Granite Code Models: A Family of Open Foundation Models for Code Intelligencecited by 116
  38. 2022ReCode: Robustness Evaluation of Code Generation Modelscited by 100

50 most cited of 85 All 85 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 62; 2025: 241; 2026: 224.

527 papers.

Sort:

By area:

  1. 2024Qwen2.5-Coder Technical Reportcited by 1,151
  2. 2024Tulu 3: Pushing Frontiers in Open Language Model Post-Trainingcited by 988
  3. 2025Phi-4-Mini Technical Report: Compact yet Powerful Multimodal Language Models via Mixture-of-LoRAscited by 491
  4. 2025Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approachcited by 372
  5. 2026SkillsBench: Benchmarking How Well Agent Skills Work Across Diverse Taskscited by 289
  6. 2025SWE-Bench Pro: Can AI Agents Solve Long-Horizon Software Engineering Tasks?cited by 254
  7. 2025SWE-RL: Advancing LLM Reasoning via Reinforcement Learning on Open Software Evolutioncited by 246
  8. 2025DeepResearch Bench: A Comprehensive Benchmark for Deep Research Agentscited by 236
  9. 2024Agent-as-a-Judge: Evaluate Agents with Agentscited by 223
  10. 2025Dynamic Early Exit in Reasoning Modelscited by 217
  11. 2024SWE-bench Multimodal: Do AI Systems Generalize to Visual Software Domains?cited by 196
  12. 2024APIGen: Automated Pipeline for Generating Verifiable and Diverse Function-Calling Datasetscited by 188
38 more
  1. 2025DiffuCoder: Understanding and Improving Masked Diffusion Models for Code Generationcited by 184
  2. 2025Seed Diffusion: A Large-Scale Diffusion Language Model with High-Speed Inferencecited by 173
  3. 2026MiMo-V2-Flash Technical Reportcited by 163
  4. 2025LLaDA2.0: Scaling Up Diffusion Language Models to 100Bcited by 154
  5. 2025R2E-Gym: Procedural Environments and Hybrid Verifiers for Scaling Open-Weights SWE Agentscited by 140
  6. 2024OpenCoder: The Open Cookbook for Top-Tier Code Large Language Modelscited by 127
  7. 2025REASONING GYM: Reasoning Environments for Reinforcement Learning with Verifiable Rewardscited by 121
  8. 2026Agentic Harness Engineering: Observability-Driven Automatic Evolution of Coding-Agent Harnessescited by 119
  9. 2025CollabLLM: From Passive Responders to Active Collaboratorscited by 117
  10. 2025KodCode: A Diverse, Challenging, and Verifiable Synthetic Dataset for Codingcited by 115
  11. 2024Scaling Laws with Vocabulary: Larger Models Deserve Larger Vocabulariescited by 113
  12. 2026MemRL: Self-Evolving Agents via Runtime Reinforcement Learning on Episodic Memorycited by 113
  13. 2025OpenCodeReasoning: Advancing Data Distillation for Competitive Codingcited by 106
  14. 2025SWE-Lancer: Can Frontier LLMs Earn 1 Million from Real-World Freelance Software Engineering?cited by 99
  15. 2024LLM Hallucinations in Practical Code Generation: Phenomena, Mechanism, and Mitigationcited by 92
  16. 2025ACECODER: Acing Coder RL via Automated Test-Case Synthesiscited by 89
  17. 2024Enhancing Code LLMs with Reinforcement Learning in Code Generation: A Surveycited by 82
  18. 2025Dream-Coder 7B: An Open Diffusion Language Model for Codecited by 81
  19. 2025BixBench: a Comprehensive Benchmark for LLM-based Agents in Computational Biologycited by 75
  20. 2026SWE-Skills-Bench: Do Agent Skills Actually Help in Real-World Software Engineering?cited by 72
  21. 2025SEC-bench: Automated Benchmarking of LLM Agents on Real-World Software Security Taskscited by 69
  22. 2025WebGen-Bench: Evaluating LLMs on Generating Interactive and Functional Websites from Scratchcited by 66
  23. 2024SelfCodeAlign: Self-Alignment for Code Generationcited by 61
  24. 2025FlowReasoner: Reinforcing Query-Level Meta-Agentscited by 56
  25. 2025lmgame-Bench: How Good are LLMs at Playing Games?cited by 54
  26. 2025LLaDA-MoE: A Sparse MoE Diffusion Language Modelcited by 54
  27. 2025Satori: Reinforcement Learning with Chain-of-Action-Thought Enhances LLM Reasoning via Autoregressive Searchcited by 53
  28. 2025DataSciBench: An LLM Agent Benchmark for Data Sciencecited by 53
  29. 2024What's Wrong with Your Code Generated by Large Language Models? An Extensive Studycited by 52
  30. 2024Commit0: Library Generation from Scratchcited by 50
  31. 2025CoT Red-Handed: Stress Testing Chain-of-Thought Monitoringcited by 49
  32. 2025SWE-EVO: Benchmarking Coding Agents in Long-Horizon Software Evolution Scenarioscited by 49
  33. 2026LLaDA2.1: Speeding Up Text Diffusion via Token Editingcited by 48
  34. 2025Code to Think, Think to Code: A Survey on Code-Enhanced Reasoning and Reasoning-Driven Code Intelligence in LLMscited by 47
  35. 2025The SWE-Bench Illusion: When State-of-the-Art LLMs Remember Instead of Reasoncited by 47
  36. 2025Retrieval Models Aren't Tool-Savvy: Benchmarking Tool Retrieval for Large Language Modelscited by 46
  37. 2025Seed-Coder: Let the Code Model Curate Data for Itselfcited by 46
  38. 2025LLM Agents Making Agent Toolscited by 45

50 most cited of 527 All 527 (JSON)