Do we agree on user interface aesthetics of Android apps?

Context: Visual aesthetics is increasingly seen as an essential factor in perceived usability, interaction, and overall appraisal of user interfaces especially with respect to mobile applications. Yet, a question that remains is how to assess and to which extend users agree on visual aesthetics. Objective: This paper analyzes the inter-rater agreement on visual aesthetics of user interfaces of Android apps as a basis for guidelines and evaluation models. Method: We systematically collected ratings on the visual aesthetics of 100 user interfaces of Android apps from 10 participants and analyzed the frequency distribution, reliability and influencing design aspects. Results: In general, user interfaces of Android apps are perceived more ugly than beautiful. Yet, raters only moderately agree on the visual aesthetics. Disagreements seem to be related to subtle differences with respect to layout, shapes, colors, typography, and background images. Conclusion: Visual aesthetics is a key factor for the success of apps. However, the considerable disagreement of raters on the perceived visual aesthetics indicates the need for a better understanding of this software quality with respect to mobile apps.