# Loose LIPS Sink Ships: Asking Questions in Battleship with Language-Informed Program Sampling

arXiv: https://arxiv.org/abs/2402.19471 · HTML: https://mlsanity.com/paper/2402.19471/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 42 references that are in the map.
- First posted on arXiv 29 Feb 2024.

## Builds on (most cited first, 16 of 16)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [Code Llama: Open Foundation Models for Code](https://mlsanity.com/paper/2308.12950/) (2023)
- [PaLM-E: An Embodied Multimodal Language Model](https://mlsanity.com/paper/2303.03378/) (2023)
- [Hypothesis Search: Inductive Reasoning with Language Models](https://mlsanity.com/paper/2309.05660/) (2023)
- [From Word Models to World Models: Translating from Natural Language to the Probabilistic Language of Thought](https://mlsanity.com/paper/2306.12672/) (2023)
- [Language Model Cascades](https://mlsanity.com/paper/2207.10342/) (2022)
- [Clarify When Necessary: Resolving Ambiguity Through Interaction with LMs](https://mlsanity.com/paper/2311.09469/) (2023)
- [Eliciting Human Preferences with Language Models](https://mlsanity.com/paper/2310.11589/) (2023)
- [Sequential Monte Carlo Steering of Large Language Models using Probabilistic Programs](https://mlsanity.com/paper/2306.03081/) (2023)
- [LILO: Learning Interpretable Libraries by Compressing and Documenting Code](https://mlsanity.com/paper/2310.19791/) (2023)
- [Leveraging Language to Learn Program Abstractions and Search Heuristics](https://mlsanity.com/paper/2106.11053/) (2021)
- [Active Preference Inference using Language Models and Probabilistic Reasoning](https://mlsanity.com/paper/2312.12009/) (2023)
- [Human-like Few-Shot Learning via Bayesian Reasoning over Natural Language](https://mlsanity.com/paper/2306.02797/) (2023)
- [Question Asking as Program Generation](https://mlsanity.com/paper/1711.06351/) (2017)

## Built on by (most cited first, 12 of 12)

- [QuestBench: Can LLMs ask the right question to acquire information in reasoning tasks?](https://mlsanity.com/paper/2503.22674/) (2025)
- [Bayesian Teaching Enables Probabilistic Reasoning in Large Language Models](https://mlsanity.com/paper/2503.17523/) (2025)
- [Learning to Ask Informative Questions: Enhancing LLMs with Preference Optimization and Expected Information Gain](https://mlsanity.com/paper/2406.17453/) (2024)
- [Ask or Assume? Uncertainty-Aware Clarification-Seeking in Coding Agents](https://mlsanity.com/paper/2603.26233/) (2026)
- [LLM-Guided Probabilistic Program Induction for POMDP Model Estimation](https://mlsanity.com/paper/2505.02216/) (2025)
- [Shoot First, Ask Questions Later? Building Rational Agents that Explore and Act Like People](https://mlsanity.com/paper/2510.20886/) (2025)
- [From Code to Play: Benchmarking Program Search for Games Using Large Language Models](https://mlsanity.com/paper/2412.04057/) (2024)
- [Goal Inference from Open-Ended Dialog](https://mlsanity.com/paper/2410.13957/) (2024)
- [Program Synthesis via Test-Time Transduction](https://mlsanity.com/paper/2509.17393/) (2025)
- [Flexible Agent Alignment with Goal Inference from Open-Ended Dialog](https://mlsanity.com/paper/2508.15119/) (2025)
- [GuessingGame: Measuring the Informativeness of Open-Ended Questions in Large Language Models](https://mlsanity.com/paper/2509.19593/) (2025)
- [Game of Thought: Robust Information Seeking with Large Language Models Using Game Theory](https://mlsanity.com/paper/2602.01708/) (2026)

## Abstract (arXiv)

Questions combine our mastery of language with our remarkable facility for reasoning about uncertainty. How do people navigate vast hypothesis spaces to pose informative questions given limited cognitive resources? We study these tradeoffs in a classic grounded question-asking task based on the board game Battleship. Our language-informed program sampling (LIPS) model uses large language models (LLMs) to generate natural language questions, translate them into symbolic programs, and evaluate their expected information gain. We find that with a surprisingly modest resource budget, this simple Monte Carlo optimization strategy yields informative questions that mirror human performance across varied Battleship board scenarios. In contrast, LLM-only baselines struggle to ground questions in the board state; notably, GPT-4V provides no improvement over non-visual baselines. Our results illustrate how Bayesian models of question-asking can leverage the statistics of language to capture human priors, while highlighting some shortcomings of pure LLMs as grounded reasoners.