Using Vision Language Foundation Models to Generate Plant Simulation Configurations via In-Context Learning
On the map: Multimodal LLMs / vision-language models → Agricultural vision-language models
Abstract (arXiv)
This paper introduces a benchmark for evaluating whether vision-language models (VLMs) can generate plant simulation configurations from imagery using in-context learning. We study this benchmark for cowpea plot reconstruction for plant simulations, where the VLM needs to generate structured JSON configurations that include field and plant information. Open-source multimodal models from Gemma 4 and Qwen3.5 families are evaluated on a synthetic cowpea dataset with known JSON ground truth and on a real drone orthophoto dataset with field-collected JSON. Five in-context learning methods are used, from format restriction instruction to few-shot image examples with auxiliary grounding information. The results show that VLMs can generate valid JSON outputs, can generally estimate days after planting (DAP), plant counts, plant locations, sun angles, and leaf chlorophyll content, and can render approximate simulations of cowpea plots. Error metrics fluctuate across model families and often remain worse than dataset baselines, particularly when VLMs' pretrained knowledge dominates over weak visual evidence. These results position image-to-simulation JSON generation as a promising but currently challenging task, and establish a benchmark for studying how multimodal reasoning, prompt design, and the sim-to-real domain gap affect plant phenotyping tasks.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Agricultural vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 13 of its 49 references that are in the map. First posted on arXiv 9 Mar 2026.
Builds on
13 of 49 references.
Closest ancestors · 3
Same direction, newest first
- 2025AgroBench: Vision-Language Model Benchmark in Agriculture28 Jul
- 2024AgroGPT: Efficient Agricultural Vision-Language Model with Expert Tuning10 Oct
- 2024Leveraging Vision Language Models for Specialized Agricultural Tasks29 Jul
Other references · 10
- 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
- 2015Show, Attend and Tell: Neural Image Caption Generation with Visual Attentioncited by 1,421
- 2021OCR-free Document Understanding Transformercited by 400
- 2022Pix2Struct: Screenshot Parsing as Pretraining for Visual Language Understandingcited by 360
- 2023Grammar-Constrained Decoding for Structured NLP Tasks without Finetuningcited by 186
- 2022DePlot: One-shot visual language reasoning by plot-to-table translationcited by 157
- 2022MatCha: Enhancing Visual Language Pretraining with Math Reasoning and Chart Derenderingcited by 132
- 2022ReCLIP: A Strong Zero-Shot Baseline for Referring Expression Comprehensioncited by 119
- 2022Prompting Is Programming: A Query Language for Large Language Modelscited by 110
- 2023VLCounter: Text-aware Visual Representation for Zero-Shot Object Countingcited by 45
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026From UAV Imagery to Agronomic Reasoning: A Multimodal LLM Benchmark for Plant Phenotyping
- 2026A Vision Language Model for Generating Procedural Plant Architecture Representations from Simulated Images
- 2026Benchmarking Vision-Language Models for Microscopic Plant Image Understanding
- 2026A Vision-Language Model (VLM)-based Pipeline for End-to-End Procedural Modeling of Field-Grown Maize from Point Clouds
- 2026Text-conditioned Segmentation for Tomato Phenotyping via Procedural Synthetic Data
- 2025AgriGPT-VL: Agricultural Vision-Language Understanding Suite
- 2025Are vision-language models ready to zero-shot replace supervised classification models in agriculture?
- 2026AgriChat: A Multimodal Large Language Model for Agriculture Image Understanding