mlsanityData to 7 Oct 2026

Which Algorithm Specification Formats Help Language Models Implement Machine Learning Algorithms?

arXiv 2607.03158 ↗First posted 3 Jul 2026cs.SE cs.AI cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Large language models (LLMs) are increasingly used to implement algorithms from research manuscripts, but papers often leave implementation choices implicit. This study examines how the written format of an algorithm specification affects first-pass LLM implementation accuracy. We compare ordinary prose, LaTeX algorithm-style pseudocode, PDF-like extracted pseudocode, Markdown fields, YAML-like specifications, JSON-like specifications, and Python code stubs across five machine learning tasks, three models, and four experimental settings, yielding 4,020 generated implementations. Hidden tests evaluate details that often determine correctness, including tie-breaking, array shapes, numerical rules, return structures, and invalid-input behavior. Under the core-information setting, LaTeX algorithm-style pseudocode has the largest average format effect, with YAML-like specifications and ordinary prose close behind. Under complete information, GPT-5.4 mini shows no format differences in the matched comparisons, whereas Gemma 3 4B and Llama 3.2 3B still do. Code stubs do not consistently improve correctness despite specifying the function signature. The results support a writing recommendation: authors should state the interface, computation steps, numerical rules, and boundary-case behavior explicitly, instead of relying on a particular surface format to carry those details.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 7 of its 15 references that are in the map. First posted on arXiv 3 Jul 2026.

Builds on

7 of 15 references.

References · 7

  1. 2021Program Synthesis with Large Language Modelscited by 3,509
  2. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,193
  3. 2025PaperBench: Evaluating AI's Ability to Replicate AI Researchcited by 286
  4. 2024Does Prompt Formatting Have Any Impact on LLM Performance?cited by 132
  5. 2025SciReplicate-Bench: Benchmarking LLMs in Agent-driven Algorithmic Reproduction from Research Paperscited by 42
  6. 2025ResearchCodeBench: Benchmarking LLMs on Implementing Novel Machine Learning Research Codecited by 34
  7. 2025From Articles to Code: On-Demand Generation of Core Algorithms from Scientific Publicationscited by 2

Built on by

Not cited yet.