# Joint Object and State Recognition using Language Knowledge

arXiv: https://arxiv.org/abs/1905.08843 · HTML: https://mlsanity.com/paper/1905.08843/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 13 May 2019.

## Built on by (most cited first, 10 of 10)

- [From Cooking Recipes to Robot Task Trees -- Improving Planning Correctness and Task Efficiency by Leveraging LLMs with a Knowledge Network](https://mlsanity.com/paper/2309.09181/) (2023)
- [Evaluating Recipes Generated from Functional Object-Oriented Network](https://mlsanity.com/paper/2106.00728/) (2021)
- [Long-Horizon Planning and Execution with Functional Object-Oriented Networks](https://mlsanity.com/paper/2207.05800/) (2022)
- [Cooking Object's State Identification Without Using Pretrained Model](https://mlsanity.com/paper/2103.02305/) (2021)
- [A Road-map to Robot Task Execution with the Functional Object-Oriented Network](https://mlsanity.com/paper/2106.00158/) (2021)
- [Functional Task Tree Generation from a Knowledge Graph to Solve Unseen Problems](https://mlsanity.com/paper/2112.02433/) (2021)
- [Calorie Aware Automatic Meal Kit Generation from an Image](https://mlsanity.com/paper/2112.09839/) (2021)
- [Task Tree Retrieval for Robotic Cooking](https://mlsanity.com/paper/2211.01745/) (2022)
- [Task Tree Retrieval Algorithms for Robotic Cooking Using The Functional Object-Oriented Network](https://mlsanity.com/paper/2211.06743/) (2022)
- [StateVLM: A State-Aware Vision-Language Model for Robotic Affordance Reasoning](https://mlsanity.com/paper/2605.03927/) (2026)

## Abstract (arXiv)

The state of an object is an important piece of knowledge in robotics applications. States and objects are intertwined together, meaning that object information can help recognize the state of an image and vice versa. This paper addresses the state identification problem in cooking related images and uses state and object predictions together to improve the classification accuracy of objects and their states from a single image. The pipeline presented in this paper includes a CNN with a double classification layer and the Concept-Net language knowledge graph on top. The language knowledge creates a semantic likelihood between objects and states. The resulting object and state confidences from the deep architecture are used together with object and state relatedness estimates from a language knowledge graph to produce marginal probabilities for objects and states. The marginal probabilities and confidences of objects (or states) are fused together to improve the final object (or state) classification results. Experiments on a dataset of cooking objects show that using a language knowledge graph on top of a deep neural network effectively enhances object and state classification.