# PAPERCLIP: Associating Astronomical Observations and Natural Language with Multi-Modal Models

arXiv: https://arxiv.org/abs/2403.08851 · HTML: https://mlsanity.com/paper/2403.08851/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 44 references that are in the map.
- First posted on arXiv 13 Mar 2024.

## Builds on (most cited first, 24 of 24)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [On the Opportunities and Risks of Foundation Models](https://mlsanity.com/paper/2108.07258/) (2021)
- [Mixtral of Experts](https://mlsanity.com/paper/2401.04088/) (2024)
- [LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale](https://mlsanity.com/paper/2208.07339/) (2022)
- [GeoCLIP: Clip-Inspired Alignment between Locations and Images for Effective Worldwide Geo-localization](https://mlsanity.com/paper/2309.16020/) (2023)
- [Towards Foundation Models for Scientific Machine Learning: Characterizing Scaling and Transfer Behavior](https://mlsanity.com/paper/2306.00258/) (2023)
- [Multiple Physics Pretraining for Physical Surrogate Models](https://mlsanity.com/paper/2310.02994/) (2023)
- [A Text-guided Protein Design Framework](https://mlsanity.com/paper/2302.04611/) (2023)
- [Masked Particle Modeling on Sets: Towards Self-Supervised High Energy Physics Foundation Models](https://mlsanity.com/paper/2401.13537/) (2024)
- [OmniJet-α: The first cross-task foundation model for particle physics](https://mlsanity.com/paper/2403.05618/) (2024)
- [Self-Supervised Representation Learning for Astronomical Images](https://mlsanity.com/paper/2012.13083/) (2020)
- [Finetuning Foundation Models for Joint Analysis Optimization](https://mlsanity.com/paper/2401.13536/) (2024)
- [Morpheus: A Deep Learning Framework For Pixel-Level Analysis of Astronomical Image Data](https://mlsanity.com/paper/1906.11248/) (2019)
- [Self-supervised similarity search for large scientific datasets](https://mlsanity.com/paper/2110.13151/) (2021)
- [Mining for Strong Gravitational Lenses with Self-supervised Learning](https://mlsanity.com/paper/2110.00023/) (2021)
- [A brief review of contrastive learning applied to astrophysics](https://mlsanity.com/paper/2306.05528/) (2023)
- [Data Compression and Inference in Cosmology with Self-Supervised Machine Learning](https://mlsanity.com/paper/2308.09751/) (2023)
- [Estimating Galactic Distances From Images Using Self-supervised Representation Learning](https://mlsanity.com/paper/2101.04293/) (2021)
- [A New Task: Deriving Semantic Class Targets for the Physical Sciences](https://mlsanity.com/paper/2210.14760/) (2022)
- [Rare Galaxy Classes Identified In Foundation Model Representations](https://mlsanity.com/paper/2312.02910/) (2023)

## Built on by (most cited first, 15 of 15)

- [From AI for Science to Agentic Science: A Survey on Autonomous Scientific Discovery](https://mlsanity.com/paper/2508.14111/) (2025)
- [A Survey of Scientific Large Language Models: From Data Foundations to Agent Frontiers](https://mlsanity.com/paper/2508.21148/) (2025)
- [FEABench: Evaluating Language Models on Multiphysics Reasoning Ability](https://mlsanity.com/paper/2504.06260/) (2025)
- [AgriCLIP: Adapting CLIP for Agriculture and Livestock via Domain-Specialized Cross-Model Alignment](https://mlsanity.com/paper/2410.01407/) (2024)
- [AstroPT: Scaling Large Observation Models for Astronomy](https://mlsanity.com/paper/2405.14930/) (2024)
- [AstroLLaVA: towards the unification of astronomical data and natural language](https://mlsanity.com/paper/2504.08583/) (2025)
- [The Platonic Universe: Do Foundation Models See the Same Sky?](https://mlsanity.com/paper/2509.19453/) (2025)
- [CosmoCLIP: Generalizing Large Vision-Language Models for Astronomical Imaging](https://mlsanity.com/paper/2407.07315/) (2024)
- [GalaxAlign: Mimicking Citizen Scientists' Multimodal Guidance for Galaxy Morphology Analysis](https://mlsanity.com/paper/2411.19475/) (2024)
- [At First Sight: Zero-Shot Classification of Astronomical Images with Large Multimodal Models](https://mlsanity.com/paper/2406.17057/) (2024)
- [A Multimodal, Multilingual, and Multidimensional Pipeline for Fine-grained Crowdsourcing Earthquake Damage Evaluation](https://mlsanity.com/paper/2506.03360/) (2025)
- [Semantic search for 100M+ galaxy images using AI-generated captions](https://mlsanity.com/paper/2512.11982/) (2025)
- [Augmenting representations with scientific papers](https://mlsanity.com/paper/2603.04516/) (2026)
- [Estimating Uncertainty in Galaxy Morphology Classification](https://mlsanity.com/paper/2608.08398/) (2026)
- [AstroSpecLM: A Spectrum-Language Model for Evidence-Grounded Astronomical Spectral Analysis](https://mlsanity.com/paper/2609.07102/) (2026)

## Abstract (arXiv)

We present PAPERCLIP (Proposal Abstracts Provide an Effective Representation for Contrastive Language-Image Pre-training), a method which associates astronomical observations imaged by telescopes with natural language using a neural network model. The model is fine-tuned from a pre-trained Contrastive Language-Image Pre-training (CLIP) model using successful observing proposal abstracts and corresponding downstream observations, with the abstracts optionally summarized via guided generation using large language models (LLMs). Using observations from the Hubble Space Telescope (HST) as an example, we show that the fine-tuned model embodies a meaningful joint representation between observations and natural language through tests targeting image retrieval (i.e., finding the most relevant observations using natural language queries) and description retrieval (i.e., querying for astrophysical object classes and use cases most relevant to a given observation). Our study demonstrates the potential for using generalist foundation models rather than task-specific models for interacting with astronomical data by leveraging text as an interface.