# OSCAR: Object Status and Contextual Awareness for Recipes to Support Non-Visual Cooking

arXiv: https://arxiv.org/abs/2503.05962 · HTML: https://mlsanity.com/paper/2503.05962/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 32 references that are in the map.
- First posted on arXiv 7 Mar 2025.

## Builds on (most cited first, 17 of 17)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Sigmoid Loss for Language Image Pre-Training](https://mlsanity.com/paper/2303.15343/) (2023)
- [VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Training](https://mlsanity.com/paper/2203.12602/) (2022)
- [CLIP4Clip: An Empirical Study of CLIP for End to End Video Clip Retrieval](https://mlsanity.com/paper/2104.08860/) (2021)
- [Towards Automatic Learning of Procedures from Web Instructional Videos](https://mlsanity.com/paper/1703.09788/) (2017)
- [Deep Learning in Video Multi-Object Tracking: A Survey](https://mlsanity.com/paper/1907.12740/) (2019)
- [Video Object Segmentation and Tracking: A Survey](https://mlsanity.com/paper/1904.09172/) (2019)
- [Temporal RoI Align for Video Object Recognition](https://mlsanity.com/paper/2109.03495/) (2021)
- [Learning Object State Changes in Videos: An Open-World Perspective](https://mlsanity.com/paper/2312.11782/) (2023)
- [A Recipe for Creating Multimodal Aligned Datasets for Sequential Tasks](https://mlsanity.com/paper/2005.09606/) (2020)
- [Temporal Action Detection with Multi-level Supervision](https://mlsanity.com/paper/2011.11893/) (2020)
- [A Contextual Inquiry of People with Vision Impairments in Cooking](https://mlsanity.com/paper/2402.15108/) (2024)
- [Non-Visual Cooking: Exploring Practices and Challenges of Meal Preparation by People with Visual Impairments](https://mlsanity.com/paper/2107.05783/) (2021)
- [A Recipe for Success? Exploring Strategies for Improving Non-Visual Access to Cooking Instructions](https://mlsanity.com/paper/2407.19065/) (2024)
- ["I Choose Assistive Devices That Save My Face" A Study on Perceptions of Accessibility and Assistive Technology Use Conducted in China](https://mlsanity.com/paper/2101.09348/) (2021)
- [Freedom to Choose: Understanding Input Modality Preferences of People with Upper-body Motor Impairments for Activities of Daily Living](https://mlsanity.com/paper/2207.04344/) (2022)

## Built on by (most cited first, 5 of 5)

- [Vid2Coach: Transforming How-To Videos into Task Assistants](https://mlsanity.com/paper/2506.00717/) (2025)
- [OSCBench: Benchmarking Object State Change in Text-to-Video Generation](https://mlsanity.com/paper/2603.11698/) (2026)
- [Exploring Object Status Recognition for Recipe Progress Tracking in Non-Visual Cooking](https://mlsanity.com/paper/2507.03330/) (2025)
- [More than One Step at a Time: Designing Procedural Feedback for Non-visual Makeup Routines](https://mlsanity.com/paper/2507.03942/) (2025)
- [How Does India Cook Biryani?](https://mlsanity.com/paper/2601.06198/) (2026)

## Abstract (arXiv)

Following recipes while cooking is an important but difficult task for visually impaired individuals. We developed OSCAR (Object Status Context Awareness for Recipes), a novel approach that provides recipe progress tracking and context-aware feedback on the completion of cooking tasks through tracking object statuses. OSCAR leverages both Large-Language Models (LLMs) and Vision-Language Models (VLMs) to manipulate recipe steps, extract object status information, align visual frames with object status, and provide cooking progress tracking log. We evaluated OSCAR's recipe following functionality using 173 YouTube cooking videos and 12 real-world non-visual cooking videos to demonstrate OSCAR's capability to track cooking steps and provide contextual guidance. Our results highlight the effectiveness of using object status to improve performance compared to baseline by over 20% across different VLMs, and we present factors that impact prediction performance. Furthermore, we contribute a dataset of real-world non-visual cooking videos with step annotations as an evaluation benchmark.