What Social Media Use Do People Regret? An Analysis of 34K Smartphone Screenshots with Multimodal LLM

Smartphone users often regret aspects of their phone use, especially social media use. However, pinpointing specific ways in which the design of an interface contributes to regrettable use can be challenging due to the complexity of social media app features and user intentions. We conducted a one-week study with 17 Android users, using a novel method where we passively collected screenshots every five seconds, which we analyzed via a multimodal large language model to understand participants' usage activity at a fine-grained level. Triangulating this data with data from experience sampling, surveys, and interviews, we found that regret varies based on user intention, with non-intentional and social media use being especially regrettable. Regret also varies by social media activity; participants were most likely to regret viewing algorithmically recommended content and comments. Additionally, participants frequently deviated to browsing social media when their intention was direct communication, which slightly increased their regret. Our findings provide guidance to designers and policy-makers seeking to improve users' experience and autonomy.