# Exploring Object Status Recognition for Recipe Progress Tracking in Non-Visual Cooking

arXiv: https://arxiv.org/abs/2507.03330 · HTML: https://mlsanity.com/paper/2507.03330/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 88 references that are in the map.
- First posted on arXiv 4 Jul 2025.

## Builds on (most cited first, 28 of 28)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Sigmoid Loss for Language Image Pre-Training](https://mlsanity.com/paper/2303.15343/) (2023)
- [VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Training](https://mlsanity.com/paper/2203.12602/) (2022)
- [CLIP4Clip: An Empirical Study of CLIP for End to End Video Clip Retrieval](https://mlsanity.com/paper/2104.08860/) (2021)
- [Towards Automatic Learning of Procedures from Web Instructional Videos](https://mlsanity.com/paper/1703.09788/) (2017)
- [Recipe1M+: A Dataset for Learning Cross-Modal Embeddings for Cooking Recipes and Food Images](https://mlsanity.com/paper/1810.06553/) (2018)
- [Benchmarking Vision Language Models for Cultural Understanding](https://mlsanity.com/paper/2407.10920/) (2024)
- [Temporal RoI Align for Video Object Recognition](https://mlsanity.com/paper/2109.03495/) (2021)
- [Learning Object State Changes in Videos: An Open-World Perspective](https://mlsanity.com/paper/2312.11782/) (2023)
- [ORBIT: A Real-World Few-Shot Dataset for Teachable Object Recognition](https://mlsanity.com/paper/2104.03841/) (2021)
- [A Recipe for Creating Multimodal Aligned Datasets for Sequential Tasks](https://mlsanity.com/paper/2005.09606/) (2020)
- [Vid2Coach: Transforming How-To Videos into Task Assistants](https://mlsanity.com/paper/2506.00717/) (2025)
- [SANPO: A Scene Understanding, Accessibility and Human Navigation Dataset](https://mlsanity.com/paper/2309.12172/) (2023)
- [Temporal Action Detection with Multi-level Supervision](https://mlsanity.com/paper/2011.11893/) (2020)
- [Understanding Visual Arts Experiences of Blind People](https://mlsanity.com/paper/2301.12687/) (2023)
- [A Contextual Inquiry of People with Vision Impairments in Cooking](https://mlsanity.com/paper/2402.15108/) (2024)
- [Non-Visual Cooking: Exploring Practices and Challenges of Meal Preparation by People with Visual Impairments](https://mlsanity.com/paper/2107.05783/) (2021)
- ["It Feels Like Taking a Gamble": Exploring Perceptions, Practices, and Challenges of Using Makeup and Cosmetics for People with Visual Impairments](https://mlsanity.com/paper/2203.08864/) (2022)
- [A Recipe for Success? Exploring Strategies for Improving Non-Visual Access to Cooking Instructions](https://mlsanity.com/paper/2407.19065/) (2024)
- [OSCAR: Object Status and Contextual Awareness for Recipes to Support Non-Visual Cooking](https://mlsanity.com/paper/2503.05962/) (2025)
- ["I Choose Assistive Devices That Save My Face" A Study on Perceptions of Accessibility and Assistive Technology Use Conducted in China](https://mlsanity.com/paper/2101.09348/) (2021)
- [TeethTap: Recognizing Discrete Teeth Gestures Using Motion and Acoustic Sensing on an Earpiece](https://mlsanity.com/paper/2102.12548/) (2021)
- [Freedom to Choose: Understanding Input Modality Preferences of People with Upper-body Motor Impairments for Activities of Daily Living](https://mlsanity.com/paper/2207.04344/) (2022)
- [Exploring the Impact of Emotional Voice Integration in Sign-to-Speech Translators for Deaf-to-Hearing Communication](https://mlsanity.com/paper/2412.05738/) (2024)
- [ThumbTrak: Recognizing Micro-finger Poses Using a Ring with Proximity Sensing](https://mlsanity.com/paper/2105.14680/) (2021)
- [Understanding How Older Adults Comprehend COVID-19 Interactive Visualizations via Think-Aloud Protocol](https://mlsanity.com/paper/2202.11441/) (2022)
- [Co-design Accessible Public Robots: Insights from People with Mobility Disability, Robotic Practitioners and Their Collaborations](https://mlsanity.com/paper/2404.05050/) (2024)

## Built on by (most cited first, 2 of 2)

- [More than One Step at a Time: Designing Procedural Feedback for Non-visual Makeup Routines](https://mlsanity.com/paper/2507.03942/) (2025)
- [Hierarchical Instance Tracking to Balance Privacy Preservation with Accessible Information](https://mlsanity.com/paper/2512.10102/) (2025)

## Abstract (arXiv)

Cooking plays a vital role in everyday independence and well-being, yet remains challenging for people with vision impairments due to limited support for tracking progress and receiving contextual feedback. Object status - the condition or transformation of ingredients and tools - offers a promising but underexplored foundation for context-aware cooking support. In this paper, we present OSCAR (Object Status Context Awareness for Recipes), a technical pipeline that explores the use of object status recognition to enable recipe progress tracking in non-visual cooking. OSCAR integrates recipe parsing, object status extraction, visual alignment with cooking steps, and time-causal modeling to support real-time step tracking. We evaluate OSCAR on 173 instructional videos and a real-world dataset of 12 non-visual cooking sessions recorded by BLV individuals in their homes. Our results show that object status consistently improves step prediction accuracy across vision-language models, and reveal key factors that impact performance in real-world conditions, such as implicit tasks, camera placement, and lighting. We contribute the pipeline of context-aware recipe progress tracking, an annotated real-world non-visual cooking dataset, and design insights to guide future context-aware assistive cooking systems.