# CookAR: Affordance Augmentations in Wearable AR to Support Kitchen Tool Interactions for People with Low Vision

arXiv: https://arxiv.org/abs/2407.13515 · HTML: https://mlsanity.com/paper/2407.13515/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 18 Jul 2024.

## Built on by (most cited first, 5 of 5)

- [Vid2Coach: Transforming How-To Videos into Task Assistants](https://mlsanity.com/paper/2506.00717/) (2025)
- [2HandedAfforder: Learning Precise Actionable Bimanual Affordances from Human Videos](https://mlsanity.com/paper/2503.09320/) (2025)
- [StreetReaderAI: Making Street View Accessible Using Context-Aware Multimodal AI](https://mlsanity.com/paper/2508.08524/) (2025)
- [Reality Proxy: Fluid Interactions with Real-World Objects in MR via Abstract Representations](https://mlsanity.com/paper/2507.17248/) (2025)
- ["Does the cafe entrance look accessible? Where is the door?" Towards Geospatial AI Agents for Visual Inquiries](https://mlsanity.com/paper/2508.15752/) (2025)

## Abstract (arXiv)

Cooking is a central activity of daily living, supporting independence as well as mental and physical health. However, prior work has highlighted key barriers for people with low vision (LV) to cook, particularly around safely interacting with tools, such as sharp knives or hot pans. Drawing on recent advancements in computer vision (CV), we present CookAR, a head-mounted AR system with real-time object affordance augmentations to support safe and efficient interactions with kitchen tools. To design and implement CookAR, we collected and annotated the first egocentric dataset of kitchen tool affordances, fine-tuned an affordance segmentation model, and developed an AR system with a stereo camera to generate visual augmentations. To validate CookAR, we conducted a technical evaluation of our fine-tuned model as well as a qualitative lab study with 10 LV participants for suitable augmentation design. Our technical evaluation demonstrates that our model outperforms the baseline on our tool affordance dataset, while our user study indicates a preference for affordance augmentations over the traditional whole object augmentations.