SuperCoder2.0: Technical Report on Exploring the feasibility of LLMs as Autonomous Programmer

arXiv 2409.11190 ↗First posted 17 Sep 2024cs.SE cs.AIcited by 3

On the map: Retrieval-augmented generation → RAG for code

Abstract (arXiv)

We present SuperCoder2.0, an advanced autonomous system designed to enhance software development through artificial intelligence. The system combines an AI-native development approach with intelligent agents to enable fully autonomous coding. Key focus areas include a retry mechanism with error output traceback, comprehensive code rewriting and replacement using Abstract Syntax Tree (ast) parsing to minimize linting issues, code embedding technique for retrieval-augmented generation, and a focus on localizing methods for problem-solving rather than identifying specific line numbers. The methodology employs a three-step hierarchical search space reduction approach for code base navigation and bug localization:utilizing Retrieval Augmented Generation (RAG) and a Repository File Level Map to identify candidate files, (2) narrowing down to the most relevant files using a File Level Schematic Map, and (3) extracting 'relevant locations' within these files. Code editing is performed through a two-part module comprising CodeGeneration and CodeEditing, which generates multiple solutions at different temperature values and replaces entire methods or classes to maintain code integrity. A feedback loop executes repository-level test cases to validate and refine solutions. Experiments conducted on the SWE-bench Lite dataset demonstrate SuperCoder2.0's effectiveness, achieving correct file localization in 84.33% of cases within the top 5 candidates and successfully resolving 34% of test instances. This performance places SuperCoder2.0 fourth globally on the SWE-bench leaderboard. The system's ability to handle diverse repositories and problem types highlights its potential as a versatile tool for autonomous software development. Future work will focus on refining the code editing process and exploring advanced embedding models for improved natural language to code mapping.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Placed in RAG for code, one of 15 directions in Retrieval-augmented generation. Builds on 16 of its 34 references that are in the map. First posted on arXiv 17 Sep 2024.

Builds on

16 of 34 references.

References · 16

2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
2021Evaluating Large Language Models Trained on Codecited by 8,053
2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,193
2024OpenHands: An Open Platform for AI Software Developers as Generalist Agentscited by 956
2022CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesiscited by 876
2023InferFix: End-to-End Program Repair with LLMscited by 113
2024From LLMs to LLM-based Agents for Software Engineering: A Survey of Current, Challenges and Futurecited by 85
2024If LLM Is the Wizard, Then Code Is the Wand: A Survey on How Code Empowers Large Language Models to Serve as Intelligent Agentscited by 77
2020Deep Learning for Source Code Modeling and Generation: Models, Applications and Challengescited by 55
2024MASAI: Modular Architecture for Software-engineering AI Agentscited by 49
2018CODIT: Code Editing with Tree-Based Neural Modelscited by 40
2024Can We Rely on LLM Agents to Draft Long-Horizon Plans? Let's Take TravelPlanner as an Examplecited by 22

4 more

2020Towards a Framework for Certification of Reliable Autonomous Systemscited by 12
2024ChatDBG: Augmenting Debugging with Large Language Modelscited by 9
2024Enhancing Trust in LLMs: Algorithms for Comparing and Interpreting LLMscited by 9
2023Bridging Code Semantic and LLMs: Semantic Chain-of-Thought Prompting for Code Generationcited by 6

Built on by

Cited by year

Citing papers per year: 2025: 3.

3 papers.

Similar, not cited

Close in wording, no citation either way