# Towards Real Smart Apps: Investigating Human-AI Interactions in Smartphone On-Device AI Apps

arXiv: https://arxiv.org/abs/2307.00756 · HTML: https://mlsanity.com/paper/2307.00756/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 16 of its 44 references that are in the map.
- First posted on arXiv 3 Jul 2023.

## Builds on (most cited first, 16 of 16)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications](https://mlsanity.com/paper/1704.04861/) (2017)
- [Conformer: Convolution-augmented Transformer for Speech Recognition](https://mlsanity.com/paper/2005.08100/) (2020)
- [SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <0.5MB model size](https://mlsanity.com/paper/1602.07360/) (2016)
- [Questioning the AI: Informing Design Practices for Explainable AI User Experiences](https://mlsanity.com/paper/2001.02478/) (2020)
- [Human-Centered Artificial Intelligence: Reliable, Safe & Trustworthy](https://mlsanity.com/paper/2002.04087/) (2020)
- [On-Device Machine Learning: An Algorithms and Learning Theory Perspective](https://mlsanity.com/paper/1911.00623/) (2019)
- [A First Look at Deep Learning Apps on Smartphones](https://mlsanity.com/paper/1812.05448/) (2018)
- [Transitioning to human interaction with AI systems: New challenges and opportunities for HCI professionals to enable human-centered AI](https://mlsanity.com/paper/2105.05424/) (2021)
- [Mind Your Weight(s): A Large-scale Study on Insufficient Machine Learning Model Protection in Mobile Apps](https://mlsanity.com/paper/2002.07687/) (2020)
- [Question-Driven Design Process for Explainable AI User Experiences](https://mlsanity.com/paper/2104.03483/) (2021)
- [Robustness of on-device Models: Adversarial Attack to Deep Learning Models on Android Apps](https://mlsanity.com/paper/2101.04401/) (2021)
- [Understanding Real-world Threats to Deep Learning Models in Android Apps](https://mlsanity.com/paper/2209.09577/) (2022)
- [Smart App Attack: Hacking Deep Learning Models in Android Apps](https://mlsanity.com/paper/2204.11075/) (2022)
- [Colors -Messengers of Concepts: Visual Design Mining for Learning Color Semantics](https://mlsanity.com/paper/1505.06532/) (2015)

## Abstract (arXiv)

With the emergence of deep learning techniques, smartphone apps are now embedded on-device AI features for enabling advanced tasks like speech translation, to attract users and increase market competitiveness. A good interaction design is important to make an AI feature usable and understandable. However, AI features have their unique challenges like sensitiveness to the input, dynamic behaviours and output uncertainty. Existing guidelines and tools either do not cover AI features or consider mobile apps which are confirmed by our informal interview with professional designers. To address these issues, we conducted the first empirical study to explore user-AI-interaction in mobile apps. We aim to understand the status of on-device AI usage by investigating 176 AI apps from 62,822 apps. We identified 255 AI features and summarised 759 implementations into three primary interaction pattern types. We further implemented our findings into a multi-faceted search-enabled gallery. The results of the user study demonstrate the usefulness of our findings.