mlsanityData to 7 Oct 2026

AstroPT: Scaling Large Observation Models for Astronomy

arXiv 2405.14930 ↗First posted 23 May 2024astro-ph.IM astro-ph.GA cs.LGcited by 9

Not on the map. Why?

Abstract (arXiv)

This work presents AstroPT, an autoregressive pretrained transformer developed with astronomical use-cases in mind. The AstroPT models presented here have been pretrained on 8.6 million 512 × 512 pixel grz-band galaxy postage stamp observations from the DESI Legacy Survey DR8. We train a selection of foundation models of increasing size from 1 million to 2.1 billion parameters, and find that AstroPT follows a similar saturating log-log scaling law to textual models. We also find that the models' performances on downstream tasks as measured by linear probing improves with model size up to the model parameter saturation point. We believe that collaborative community development paves the best route towards realising an open source `Large Observation Model' -- a model trained on data taken from the observational sciences at the scale seen in natural language processing. To this end, we release the source code, weights, and dataset for AstroPT under the MIT license, and invite potential collaborators to join us in collectively building and researching these models.

Key facts. Cited by 9 papers in the map (data through 7 Oct 2026). Builds on 51 of its 82 references that are in the map. First posted on arXiv 23 May 2024.

Builds on

51 of 82 references.

References · 51

  1. 2017Attention Is All You Needcited by 59,932
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  6. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  7. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  8. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  9. 2020Scaling Laws for Neural Language Modelscited by 6,411
  10. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  11. 2015Domain-Adversarial Training of Neural Networkscited by 3,023
  12. 2022Training Compute-Optimal Large Language Modelscited by 2,927
38 more
  1. 2018UMAP: Uniform Manifold Approximation and Projection for Dimension Reductioncited by 2,841
  2. 2022Emergent Abilities of Large Language Modelscited by 2,483
  3. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,734
  4. 2019Energy and Policy Considerations for Deep Learning in NLPcited by 1,242
  5. 2021VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learningcited by 1,013
  6. 2016Conditional Image Generation with PixelCNN Decoderscited by 859
  7. 2016Pixel Recurrent Neural Networkscited by 814
  8. 2023RWKV: Reinventing RNNs for the Transformer Eracited by 814
  9. 2022A Generalist Agentcited by 802
  10. 2024TinyLlama: An Open-Source Small Language Modelcited by 604
  11. 2023Textbooks Are All You Needcited by 561
  12. 2020Scaling Laws for Autoregressive Generative Modelingcited by 510
  13. 2017PixelCNN++: Improving the PixelCNN with Discretized Logistic Mixture Likelihood and Other Modificationscited by 415
  14. 2024The Platonic Representation Hypothesiscited by 362
  15. 2022Compute Trends Across Three Eras of Machine Learningcited by 161
  16. 2024Eagle and Finch: RWKV with Matrix-Valued States and Dynamic Recurrencecited by 155
  17. 2023Scaling Laws for Generative Mixed-Modal Language Modelscited by 131
  18. 2017Fader Networks: Manipulating Images by Sliding Attributescited by 118
  19. 2024Scalable Pre-training of Large Autoregressive Image Modelscited by 115
  20. 2023LLM360: Towards Fully Transparent Open-Source LLMscited by 100
  21. 2023To Repeat or Not To Repeat: Insights from Scaling LLM under Token-Crisiscited by 88
  22. 2023AstroLLaMA: Towards Specialized Foundation Models in Astronomycited by 41
  23. 2023Scaling MLPs: A Tale of Inductive Biascited by 37
  24. 2021Realistic galaxy image simulation via score-based generative modelscited by 24
  25. 2024Application-Driven Innovation in Machine Learningcited by 24
  26. 2020Self-Supervised Representation Learning for Astronomical Imagescited by 18
  27. 2022Astronomia ex machina: a history, primer, and outlook on neural networks in astronomycited by 17
  28. 2023ConvNets Match Vision Transformers at Scalecited by 16
  29. 2024PAPERCLIP: Associating Astronomical Observations and Natural Language with Multi-Modal Modelscited by 15
  30. 2021Practical Galaxy Morphology Tools from Deep Supervised Representation Learningcited by 14
  31. 2022EleutherAI: Going Beyond "Open Science" to "Science in the Open"cited by 13
  32. 2024AstroLLaMA-Chat: Scaling AstroLLaMA with Conversational and Diverse Datasetscited by 13
  33. 2020Domain adaptation techniques for improved cross-domain study of galaxy mergerscited by 7
  34. 2013SKYNET: an efficient and robust neural network training tool for machine learning in astronomycited by 6
  35. 2023A brief review of contrastive learning applied to astrophysicscited by 5
  36. 2023Data Compression and Inference in Cosmology with Self-Supervised Machine Learningcited by 5
  37. 2024Scaling Laws for Galaxy Imagescited by 5
  38. 2018Exploring galaxy evolution with generative modelscited by 4

50 most cited of 51 All 51 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 5; 2026: 4.

9 papers.

  1. 2025From AI for Science to Agentic Science: A Survey on Autonomous Scientific Discoverycited by 74
  2. 2025A Survey of Scientific Large Language Models: From Data Foundations to Agent Frontierscited by 34
  3. 2025The Platonic Universe: Do Foundation Models See the Same Sky?cited by 5
  4. 2025Universal Spectral Tokenization via Self-Supervised Panchromatic Representation Learningcited by 2
  5. 2025Dargana: fine-tuning EarthPT for dynamic tree canopy mapping from spacecited by 1
  6. 2026Generalization from Low- to Moderate-Resolution Spectra with Neural Networks for Stellar Parameter Estimation: A Case Study with DESIcited by 0
  7. 2026What AstroPT knows about galaxies, and what that can teach us about LLMscited by 0
  8. 2026A survey detection channel overrides the pixels in an astronomical foundation model, and biases tomographic mean redshiftscited by 0
  9. 2026Gestalt: a meta-foundation model for astronomycited by 0