mlsanityData to 7 Oct 2026

SWE-Lancer: Can Frontier LLMs Earn 1 Million from Real-World Freelance Software Engineering?

arXiv 2502.12115 ↗First posted 17 Feb 2025cs.LG cs.SEcited by 99

Not on the map. Why?

Cited by 56 in LLM agents and tool use, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We introduce SWE-Lancer, a benchmark of over 1,400 freelance software engineering tasks from Upwork, valued at 1 million USD total in real-world payouts. SWE-Lancer encompasses both independent engineering tasks--ranging from 50 bug fixes to 32,000 feature implementations--and managerial tasks, where models choose between technical implementation proposals. Independent tasks are graded with end-to-end tests triple-verified by experienced software engineers, while managerial decisions are assessed against the choices of the original hired engineering managers. We evaluate model performance and find that frontier models are still unable to solve the majority of tasks. To facilitate future research, we open-source a unified Docker image and a public evaluation split, SWE-Lancer Diamond (https://github.com/openai/SWELancer-Benchmark). By mapping model performance to monetary value, we hope SWE-Lancer enables greater research into the economic impact of AI model development.

Key facts. Cited by 99 papers in the map (data through 7 Oct 2026). Builds on 16 of its 52 references that are in the map. First posted on arXiv 17 Feb 2025.

Builds on

16 of 52 references.

References · 16

  1. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  2. 2021Program Synthesis with Large Language Modelscited by 3,509
  3. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,193
  4. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 1,906
  5. 2022Competition-Level Code Generation with AlphaCodecited by 1,427
  6. 2021Measuring Coding Challenge Competence With APPScited by 879
  7. 2022CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesiscited by 876
  8. 2021CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generationcited by 732
  9. 2024BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructionscited by 527
  10. 2024Can LLMs Generate Novel Research Ideas? A Large-Scale Human Study with 100+ NLP Researcherscited by 348
  11. 2024SWE-bench Multimodal: Do AI Systems Generalize to Visual Software Domains?cited by 196
  12. 2024SciCode: A Research Coding Benchmark Curated by Scientistscited by 172
4 more
  1. 2025CodeElo: Benchmarking Competition-level Code Generation of LLMs with Human-comparable Elo Ratingscited by 109
  2. 2024Concept Induction: Analyzing Unstructured Text with High-Level Concepts Using LLooMcited by 77
  3. 2023Goal Driven Discovery of Distributional Differences via Language Descriptionscited by 61
  4. 2024NaturalCodeBench: Examining Coding Performance Mismatch on HumanEval and Natural User Promptscited by 19

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 49; 2026: 50.

99 papers.

Sort:

By area:

  1. 2025Kimi K2: Open Agentic Intelligencecited by 523
  2. 2025Survey on Evaluation of LLM-based Agentscited by 190
  3. 2025Multi-SWE-bench: A Multilingual Benchmark for Issue Resolvingcited by 165
  4. 2025From LLM Reasoning to Autonomous AI Agents: A Comprehensive Reviewcited by 146
  5. 2025Measuring AI Ability to Complete Long Software Taskscited by 142
  6. 2025GDPval: Evaluating AI Model Performance on Real-World Economically Valuable Taskscited by 142
  7. 2026Agentic Harness Engineering: Observability-Driven Automatic Evolution of Coding-Agent Harnessescited by 119
  8. 2025Establishing Best Practices for Building Rigorous Agentic Benchmarkscited by 94
  9. 2025Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivitycited by 93
  10. 2025SWE-PolyBench: A multi-language benchmark for repository level evaluation of coding agentscited by 80
  11. 2025WebGen-Bench: Evaluating LLMs on Generating Interactive and Functional Websites from Scratchcited by 66
  12. 2025OpenCodeInstruct: A Large-scale Instruction Tuning Dataset for Code LLMscited by 57
38 more
  1. 2025MLR-Bench: Evaluating AI Agents on Open-Ended Machine Learning Researchcited by 57
  2. 2025NL2Repo-Bench: Towards Long-Horizon Repository Generation Evaluation of Coding Agentscited by 56
  3. 2026Code as Agent Harnesscited by 40
  4. 2025Z1: Efficient Test-time Scaling with Codecited by 35
  5. 2025Cost-of-Pass: An Economic Framework for Evaluating Language Modelscited by 35
  6. 2025ResearcherBench: Evaluating Deep AI Research Systems on the Frontiers of Scientific Inquirycited by 30
  7. 2025PRBench: Large-Scale Expert Rubrics for Evaluating High-Stakes Professional Reasoningcited by 29
  8. 2026AgencyBench: Benchmarking the Frontiers of Autonomous Agents in 1M-Token Real-World Contextscited by 26
  9. 2025ViCrit: A Verifiable Reinforcement Learning Proxy Task for Visual Perception in VLMscited by 25
  10. 2025GitTaskBench: A Benchmark for Code Agents Solving Real-World Tasks Through Code Repository Leveragingcited by 25
  11. 2026Stop Comparing LLM Agents Without Disclosing the Harnesscited by 25
  12. 2025GSO: Challenging Software Optimization Tasks for Evaluating SWE-Agentscited by 24
  13. 2025CoRe: Benchmarking LLMs Code Reasoning Capabilities through Static Analysis Taskscited by 24
  14. 2026ContextBench: A Benchmark for Context Retrieval in Coding Agentscited by 23
  15. 2025Agentic AI Software Engineers: Programming with Trustcited by 19
  16. 2025HCAST: Human-Calibrated Autonomy Software Taskscited by 19
  17. 2025Remote Labor Index: Measuring AI Automation of Remote Workcited by 18
  18. 2025SWE-SQL: Illuminating LLM Pathways to Solve User SQL Issues in Real-World Applicationscited by 15
  19. 2025From Code Foundation Models to Agents and Applications: A Comprehensive Survey and Practical Guide to Code Intelligencecited by 13
  20. 2026AgentLens: Revealing The Lucky Pass Problem in SWE-Agent Evaluationcited by 13
  21. 2025SPICE: An Automated SWE-Bench Labeling Pipeline for Issue Clarity, Test Coverage, and Effort Estimationcited by 11
  22. 2025You Don't Know Until You Click:Automated GUI Testing for Production-Ready Software Evaluationcited by 11
  23. 2025WebGen-Agent: Enhancing Interactive Website Generation with Multi-Level Feedback and Step-Level Reinforcement Learningcited by 11
  24. 2026How Well Does Agent Development Reflect Real-World Work?cited by 10
  25. 2026Automated Benchmark Auditing for AI Agents and Large Language Modelscited by 10
  26. 2025Can Agents Fix Agent Issues?cited by 8
  27. 2025Control Tax: The Price of Keeping AI in Checkcited by 7
  28. 2025InfoAgent: Advancing Autonomous Information-Seeking Agentscited by 7
  29. 2026Vibe Code Bench: Evaluating AI Models on End-to-End Web Application Developmentcited by 7
  30. 2025Agentic Program Repair from Test Failures at Scale: A Neuro-symbolic approach with static analysis and test execution feedbackcited by 6
  31. 2026OccuBench: Evaluating AI Agents on Real-World Professional Tasks via Language Environment Simulationcited by 6
  32. 2026BigFinanceBench: A Workflow-Grounded Benchmark for Financial-Research Agentscited by 6
  33. 2025Turning the Tide: Repository-based Code Reflectioncited by 5
  34. 2026MatrAIx: Simulating the World with 8.3 Billion Persona Agentscited by 5
  35. 2025Breakpoint: Scalable evaluation of system-level reasoning in LLM code agentscited by 4
  36. 2025Evaluating LLM Generated Detection Rules in Cybersecuritycited by 4
  37. 2026ServImage: An Image Generation and Editing Benchmark from Real-world Commercial Imaging Servicescited by 4
  38. 2026CEO-Bench: Can Agents Play the Long Game?cited by 4

50 most cited of 99 All 99 (JSON)