Themisto: Jupyter-Based Runtime Benchmark
Not on the map. Why?
Abstract (arXiv)
In this work, we present a benchmark that consists of Jupyter notebooks development trajectories and allows measuring how large language models (LLMs) can leverage runtime information for predicting code output and code generation. We demonstrate that the current generation of LLMs performs poorly on these tasks and argue that there exists a significantly understudied domain in the development of code-based models, which involves incorporating the runtime context.
Key facts. No citing papers in the map yet (data through 13 Oct 2026). Builds on 15 of its 19 references that are in the map. First posted on arXiv 16 Apr 2025.
Builds on
15 of 19 references.
References · 15
- 2021Evaluating Large Language Models Trained on Codecited by 8,205
- 2024GPT-4o System Cardcited by 4,536
- 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,274
- 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,197
- 2024A Survey on Large Language Models for Code Generationcited by 656
- 2024CRUXEval: A Benchmark for Code Reasoning, Understanding and Executioncited by 297
- 2023LEVER: Learning to Verify Language-to-Code Generation with Executioncited by 235
- 2024RLEF: Grounding Code LLMs in Execution Feedback with Reinforcement Learningcited by 150
- 2023RLTF: Reinforcement Learning from Unit Test Feedbackcited by 112
- 2024StepCoder: Improve Code Generation with Reinforcement Learning from Compiler Feedbackcited by 75
- 2022Out of the BLEU: how should we assess quality of the Code Generation models?cited by 65
- 2024Reasoning Runtime Behavior of a Program with LLM: How Far Are We?cited by 44
3 more
- 2023Code Execution with Pre-trained Language Modelscited by 34
- 2023TRACED: Execution-aware Pre-training for Source Codecited by 27
- 2021KGTorrent: A Dataset of Python Jupyter Notebooks from Kagglecited by 10
Built on by
Not cited yet.