# Semantic Parsing to Probabilistic Programs for Situated Question Answering

arXiv: https://arxiv.org/abs/1606.07046 · HTML: https://mlsanity.com/paper/1606.07046/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 22 Jun 2016.

## Built on by (most cited first, 5 of 5)

- [Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering](https://mlsanity.com/paper/2209.09513/) (2022)
- [Do Vision-Language Models Really Understand Visual Language?](https://mlsanity.com/paper/2410.00193/) (2024)
- [Chimera: Diagnosing Shortcut Learning in Visual-Language Understanding](https://mlsanity.com/paper/2509.22437/) (2025)
- [Constructing Multimodal Datasets from Scratch for Rapid Development of a Japanese Visual Language Model](https://mlsanity.com/paper/2410.22736/) (2024)
- [Hybrid Primal Sketch: Combining Analogy, Qualitative Representations, and Computer Vision for Scene Understanding](https://mlsanity.com/paper/2407.04859/) (2024)

## Abstract (arXiv)

Situated question answering is the problem of answering questions about an environment such as an image or diagram. This problem requires jointly interpreting a question and an environment using background knowledge to select the correct answer. We present Parsing to Probabilistic Programs (P3), a novel situated question answering model that can use background knowledge and global features of the question/environment interpretation while retaining efficient approximate inference. Our key insight is to treat semantic parses as probabilistic programs that execute nondeterministically and whose possible executions represent environmental uncertainty. We evaluate our approach on a new, publicly-released data set of 5000 science diagram questions, outperforming several competitive classical and neural baselines.