mlsanityData to 14 Oct 2026

RepoST: Scalable Repository-Level Coding Environment Construction with Sandbox Testing

arXiv 2503.07358 ↗First posted 10 Mar 2025cs.CL cs.SEcited by 13

Not on the map. Why?

Abstract (arXiv)

We present RepoST, a scalable method to construct environments that provide execution feedback for repository-level code generation for both training and evaluation. Unlike existing works that aim to build entire repositories for execution, which is challenging for both human and LLMs, we provide execution feedback with sandbox testing, which isolates a given target function and its dependencies to a separate script for testing. Sandbox testing reduces the complexity of external dependencies and enables constructing environments at a large scale. We use our method to construct RepoST-Train, a large-scale train set with 7,415 functions from 832 repositories. Training with the execution feedback provided by RepoST-Train leads to a performance gain of 5.5% Pass@1 on HumanEval and 3.5% Pass@1 on RepoEval. We also build an evaluation dataset, RepoST-Eval, and benchmark 12 code generation models.

Key facts. Cited by 13 papers in the map (data through 14 Oct 2026). Builds on 25 of its 33 references that are in the map. First posted on arXiv 10 Mar 2025.

Builds on

25 of 33 references.

References · 25

  1. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  2. 2021Program Synthesis with Large Language Modelscited by 3,637
  3. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,373
  4. 2023Code Llama: Open Foundation Models for Codecited by 2,139
  5. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 2,004
  6. 2023Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generationcited by 1,391
  7. 2024DeepSeek-Coder: When the Large Language Model Meets Programming -- The Rise of Code Intelligencecited by 1,181
  8. 2023Teaching Large Language Models to Self-Debugcited by 988
  9. 2021Measuring Coding Challenge Competence With APPScited by 902
  10. 2023WizardCoder: Empowering Code Large Language Models with Evol-Instructcited by 733
  11. 2024StarCoder 2 and The Stack v2: The Next Generationcited by 502
  12. 2022DS-1000: A Natural and Reliable Benchmark for Data Science Code Generationcited by 452
13 more
  1. 2023RepoCoder: Repository-Level Code Completion Through Iterative Retrieval and Generationcited by 367
  2. 2024Training Software Engineering Agents and Verifiers with SWE-Gymcited by 336
  3. 2020Unsupervised Translation of Programming Languagescited by 263
  4. 2023ClassEval: A Manually-Crafted Benchmark for Evaluating LLMs on Class-level Code Generationcited by 168
  5. 2022MultiPL-E: A Scalable and Extensible Approach to Benchmarking Neural Code Generationcited by 136
  6. 2024CodeRAG-Bench: Can Retrieval Augment Code Generation?cited by 91
  7. 2024NExT: Teaching Large Language Models to Reason about Code Executioncited by 63
  8. 2024SUPER: Evaluating Agents on Setting Up and Executing Tasks from Research Repositoriescited by 52
  9. 2024TestGenEval: A Real World Unit Test Generation and Test Completion Benchmarkcited by 52
  10. 2024LeDex: Training LLMs to Better Self-Debug and Explain Codecited by 40
  11. 2024EvoCodeBench: An Evolving Code Generation Benchmark with Domain-Specific Evaluationscited by 34
  12. 2024DevEval: Evaluating Code Generation in Practical Software Projectscited by 9
  13. 2024PLUM: Improving Code LMs with Execution-Guided On-Policy Preference Learning Driven By Synthetic Test Casescited by 8

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 6; 2026: 7.

13 papers.

Sort:

By area:

  1. 2025SWE-smith: Scaling Data for Software Engineering Agentscited by 285
  2. 2025SWE-rebench: An Automated Pipeline for Task Collection and Decontaminated Evaluation of Software Engineering Agentscited by 142
  3. 2025GSO: Challenging Software Optimization Tasks for Evaluating SWE-Agentscited by 25
  4. 2025From Code Foundation Models to Agents and Applications: A Comprehensive Survey and Practical Guide to Code Intelligencecited by 15
  5. 2025AutoSDT: Scaling Data-Driven Discovery Tasks Toward Open Co-Scientistscited by 11
  6. 2026SERA: Soft-Verified Efficient Repository Agentscited by 10
  7. 2026Hybrid-Gym: Training Coding Agents to Generalize Across Taskscited by 6
  8. 2025Gistify! Codebase-Level Understanding via Runtime Executioncited by 4
  9. 2026Constraint Decay: The Fragility of LLM Agents in Backend Code Generationcited by 3
  10. 2026D3-Gym: Constructing Real-World Verifiable Environments for Data-Driven Discoverycited by 1
  11. 2026BootstrapAgent: Distilling Repository Setup into Reusable Agent Knowledgecited by 0
  12. 2026Adaptive Critical Token-Aware Retrieval for Repository-Level Code Generationcited by 0
1 more
  1. 2026SWE-Stream: Scaling Executable Training Data for SWE Agents via Progressive Construction and Elastic Streamingcited by 0