# Predicate Hierarchies Improve Few-Shot State Classification

arXiv: https://arxiv.org/abs/2502.12481 · HTML: https://mlsanity.com/paper/2502.12481/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 39 of its 51 references that are in the map.
- First posted on arXiv 18 Feb 2025.

## Builds on (most cited first, 30 of 39)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models](https://mlsanity.com/paper/2301.12597/) (2023)
- [FiLM: Visual Reasoning with a General Conditioning Layer](https://mlsanity.com/paper/1709.07871/) (2017)
- [Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answering](https://mlsanity.com/paper/1707.07998/) (2017)
- [MDETR -- Modulated Detection for End-to-End Multi-Modal Understanding](https://mlsanity.com/paper/2104.12763/) (2021)
- [Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMs](https://mlsanity.com/paper/2401.06209/) (2024)
- [SpatialVLM: Endowing Vision-Language Models with Spatial Reasoning Capabilities](https://mlsanity.com/paper/2401.12168/) (2024)
- [Poincar'e Embeddings for Learning Hierarchical Representations](https://mlsanity.com/paper/1705.08039/) (2017)
- [ViperGPT: Visual Inference via Python Execution for Reasoning](https://mlsanity.com/paper/2303.08128/) (2023)
- [CALVIN: A Benchmark for Language-Conditioned Policy Learning for Long-Horizon Robot Manipulation Tasks](https://mlsanity.com/paper/2112.03227/) (2021)
- [Extract Free Dense Labels from CLIP](https://mlsanity.com/paper/2112.01071/) (2021)
- [Hyperbolic Neural Networks](https://mlsanity.com/paper/1805.09112/) (2018)
- [Neural Module Networks](https://mlsanity.com/paper/1511.02799/) (2015)
- [How Much Can CLIP Benefit Vision-and-Language Tasks?](https://mlsanity.com/paper/2107.06383/) (2021)
- [Neural-Symbolic VQA: Disentangling Reasoning from Vision and Language Understanding](https://mlsanity.com/paper/1810.02338/) (2018)
- [Towards Open World Object Detection](https://mlsanity.com/paper/2103.02603/) (2021)
- [Learning Continuous Hierarchies in the Lorentz Model of Hyperbolic Geometry](https://mlsanity.com/paper/1806.03417/) (2018)
- [Towards Open World Recognition](https://mlsanity.com/paper/1412.5687/) (2014)
- [Hyperbolic Image Embeddings](https://mlsanity.com/paper/1904.02239/) (2019)
- [Exploring Visual Relationship for Image Captioning](https://mlsanity.com/paper/1809.07041/) (2018)
- [Representation Tradeoffs for Hyperbolic Embeddings](https://mlsanity.com/paper/1804.03329/) (2018)
- [Detecting and Recognizing Human-Object Interactions](https://mlsanity.com/paper/1704.07333/) (2017)
- [Poincar'e GloVe: Hyperbolic Word Embeddings](https://mlsanity.com/paper/1810.06546/) (2018)
- [Hyperbolic Image-Text Representations](https://mlsanity.com/paper/2304.09172/) (2023)
- [Geoopt: Riemannian Optimization in PyTorch](https://mlsanity.com/paper/2005.02819/) (2020)
- [Hyperbolic Vision Transformers: Combining Improvements in Metric Learning](https://mlsanity.com/paper/2203.10833/) (2022)
- [Hyperbolic Image Segmentation](https://mlsanity.com/paper/2203.05898/) (2022)
- [Semantic Abstraction: Open-World 3D Scene Understanding from 2D Vision-Language Models](https://mlsanity.com/paper/2207.11514/) (2022)
- [From Trees to Continuous Embeddings and Back: Hyperbolic Hierarchical Clustering](https://mlsanity.com/paper/2010.00402/) (2020)

## Built on by (most cited first, 1 of 1)

- [Explain Before You Answer: A Survey on Compositional Visual Reasoning](https://mlsanity.com/paper/2508.17298/) (2025)

## Abstract (arXiv)

State classification of objects and their relations is core to many long-horizon tasks, particularly in robot planning and manipulation. However, the combinatorial explosion of possible object-predicate combinations, coupled with the need to adapt to novel real-world environments, makes it a desideratum for state classification models to generalize to novel queries with few examples. To this end, we propose PHIER, which leverages predicate hierarchies to generalize effectively in few-shot scenarios. PHIER uses an object-centric scene encoder, self-supervised losses that infer semantic relations between predicates, and a hyperbolic distance metric that captures hierarchical structure; it learns a structured latent space of image-predicate pairs that guides reasoning over state classification queries. We evaluate PHIER in the CALVIN and BEHAVIOR robotic environments and show that PHIER significantly outperforms existing methods in few-shot, out-of-distribution state classification, and demonstrates strong zero- and few-shot generalization from simulated to real-world tasks. Our results demonstrate that leveraging predicate hierarchies improves performance on state classification tasks with limited data.