Reem Omair

Artificial Intelligence Specialization · Cohort 2026

Prompt or Train: Quantifying the Labeled-Data Trade-Off Between Fine-Tuned Models and Vision-Language Models in Astronomy

Modern astronomy produces far more images than can realistically be labeled by humans. Even large citizen-science efforts such as Galaxy Zoo have required years of volunteer effort to label only ~million galaxies, while reliable expert labels for rare astronomical objects are far scarcer. At the same time, vision-language models (VLMs) can perform image-classification tasks directly from text prompts, without task-specific training. This raises a practical question: when is it worth investing in labeled data and specialized training, and when is prompting a general-purpose VLM already sufficient? In this project, we establish a strong specialist baseline by fine-tuning leading astronomy-pretrained vision foundation models on new astronomical vision tasks, and compare their performance with VLMs such as Gemini under different prompting setups. We systematically reduce the amount of labeled data available for fine-tuning—from tens of thousands of images to only a few dozen—to measure how much data is required for specialized models to justify the effort of labeling and training them compared with prompting VLMs. By comparing these approaches across tasks of varying difficulty and different setups, we aim to quantify how the value of labeled data changes relative to the capabilities of general-purpose VLMs.

Read the final report (PDF)