# Rethinking Cooking State Recognition with Vision Transformers

arXiv: https://arxiv.org/abs/2212.08586 · HTML: https://mlsanity.com/paper/2212.08586/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 33 references that are in the map.
- First posted on arXiv 16 Dec 2022.

## Builds on (most cited first, 21 of 21)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016)
- [Rethinking the Inception Architecture for Computer Vision](https://mlsanity.com/paper/1512.00567/) (2015)
- [MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications](https://mlsanity.com/paper/1704.04861/) (2017)
- [Transformers in Vision: A Survey](https://mlsanity.com/paper/2101.01169/) (2021)
- [Quantifying Attention Flow in Transformers](https://mlsanity.com/paper/2005.00928/) (2020)
- [Do Vision Transformers See Like Convolutional Neural Networks?](https://mlsanity.com/paper/2108.08810/) (2021)
- [Albumentations: fast and flexible image augmentations](https://mlsanity.com/paper/1809.06839/) (2018)
- [A Survey of Convolutional Neural Networks: Analysis, Applications, and Prospects](https://mlsanity.com/paper/2004.02806/) (2020)
- [What's Cookin'? Interpreting Cooking Videos using Text, Speech and Vision](https://mlsanity.com/paper/1503.01558/) (2015)
- [Less is More: Lighter and Faster Deep Neural Architecture for Tomato Leaf Disease Classification](https://mlsanity.com/paper/2109.02394/) (2021)
- [Two Decades of Bengali Handwritten Digit Recognition: A Survey](https://mlsanity.com/paper/2206.02234/) (2022)
- [Fruit Quality Assessment with Densely Connected Convolutional Neural Network](https://mlsanity.com/paper/2212.04255/) (2022)
- [Identifying Object States in Cooking-Related Images](https://mlsanity.com/paper/1805.06956/) (2018)
- [Classifying cooking object's state using a tuned VGG convolutional neural network](https://mlsanity.com/paper/1805.09391/) (2018)
- [Real-time Bangla License Plate Recognition System for Low Resource Video-based Applications](https://mlsanity.com/paper/2108.08339/) (2021)
- [Cooking State Recognition from Images Using Inception Architecture](https://mlsanity.com/paper/1805.09967/) (2018)
- [VGG Fine-tuning for Cooking State Recognition](https://mlsanity.com/paper/1905.08606/) (2019)

## Built on by (most cited first, 9 of 9)

- [AQUA20: A Benchmark Dataset for Underwater Species Classification under Challenging Conditions](https://mlsanity.com/paper/2506.17455/) (2025)
- [The Multimodal And Modular Ai Chef: Complex Recipe Generation From Imagery](https://mlsanity.com/paper/2304.02016/) (2023)
- [An Efficient Transfer Learning-based Approach for Apple Leaf Disease Classification](https://mlsanity.com/paper/2304.06520/) (2023)
- [MangoLeafViT: Leveraging Lightweight Vision Transformer with Runtime Augmentation for Efficient Mango Leaf Disease Classification](https://mlsanity.com/paper/2505.23961/) (2025)
- [DExNet: Combining Observations of Domain Adapted Critics for Leaf Disease Classification with Limited Data](https://mlsanity.com/paper/2506.18173/) (2025)
- [Real-World Cooking Robot System from Recipes Based on Food State Recognition Using Foundation Models and PDDL](https://mlsanity.com/paper/2410.02874/) (2024)
- [SpectroFusion-ViT: A Lightweight Transformer for Speech Emotion Recognition Using Harmonic Mel-Chroma Fusion](https://mlsanity.com/paper/2603.00746/) (2026)
- [Few-Shot Learning Pipeline for Monkeypox Skin Disease Classification Using CNN Feature Extractors](https://mlsanity.com/paper/2605.05034/) (2026)
- [Beyond Accuracy: A Qualitative Analysis of Vision-Language Models for Hate Speech Detection in Memes](https://mlsanity.com/paper/2608.26143/) (2026)

## Abstract (arXiv)

To ensure proper knowledge representation of the kitchen environment, it is vital for kitchen robots to recognize the states of the food items that are being cooked. Although the domain of object detection and recognition has been extensively studied, the task of object state classification has remained relatively unexplored. The high intra-class similarity of ingredients during different states of cooking makes the task even more challenging. Researchers have proposed adopting Deep Learning based strategies in recent times, however, they are yet to achieve high performance. In this study, we utilized the self-attention mechanism of the Vision Transformer (ViT) architecture for the Cooking State Recognition task. The proposed approach encapsulates the globally salient features from images, while also exploiting the weights learned from a larger dataset. This global attention allows the model to withstand the similarities between samples of different cooking objects, while the employment of transfer learning helps to overcome the lack of inductive bias by utilizing pretrained weights. To improve recognition accuracy, several augmentation techniques have been employed as well. Evaluation of our proposed framework on the `Cooking State Recognition Challenge Dataset' has achieved an accuracy of 94.3%, which significantly outperforms the state-of-the-art.