mlsanityData to 7 Oct 2026

Quantifying the Reconstructability of Astrophysical Methods with Large Language Models and Information Theory: A Case Study in Spectral Reconstruction

arXiv 2605.11154 ↗First posted 11 May 2026astro-ph.IM cs.AI cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Modern astrophysical studies rely heavily on complex data analysis pipelines; however, published descriptions often lack the detail required for computational reproducibility. In this work, we present an information-theoretic framework to quantify how effectively a method can be reconstructed from its written description. By treating algorithmic reconstruction as a probability distribution generated by Large Language Models (LLMs), we utilize Shannon entropy and Jensen-Shannon divergence to measure how strongly text constrains the hypothesis space of valid implementations. We demonstrate this approach through a case study of Trans-Neptunian Object (TNO) spectral reconstruction from sparse photometry. By prompting frontier LLMs with varying levels of manuscript text (Title, Abstract, and Methods), we find that while increasing text successfully clarifies the overall algorithmic structure, it fails to eliminate variance at the implementation level. This persistent variance establishes an "entropy floor," demonstrating that multiple divergent implementations remain consistent with explicit instructions. To evaluate practical reproducibility, we convert these reconstructed algorithms into executable pipelines. Our results reveal that, while LLMs easily recover core functional methodologies, they systematically fail to infer the tacit expert knowledge required for strict scientific calibration. This pilot study demonstrates that LLMs can be repurposed as a zero-shot diagnostic tool to audit methodological transparency, helping authors identify missing structural constraints and preserve scientific integrity in an era of automated research.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 8 of its 27 references that are in the map. First posted on arXiv 11 May 2026.

Builds on

8 of 27 references.

References · 8

  1. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  2. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,146
  3. 2018UMAP: Uniform Manifold Approximation and Projection for Dimension Reductioncited by 2,841
  4. 2022Competition-Level Code Generation with AlphaCodecited by 1,427
  5. 2025Paper2Code: Automating Code Generation from Scientific Papers in Machine Learningcited by 67
  6. 2013Practices in source code sharing in astrophysicscited by 4
  7. 2018Schroedinger's code: A preliminary study on research source code availability and link persistence in astrophysicscited by 2
  8. 2026What Understanding Means in AI-Laden Astronomycited by 2

Built on by

Not cited yet.