mlsanityData to 14 Oct 2026

Computer User Interface Understanding. A New Dataset and a Learning Framework

arXiv 2403.10170 ↗First posted 15 Mar 2024cs.CVcited by 0

Not on the map. Why?

Abstract (arXiv)

User Interface (UI) understanding has been an increasingly popular topic over the last few years. So far, there has been a vast focus solely on web and mobile applications. In this paper, we introduce the harder task of computer UI understanding. With the goal of enabling research in this field, we have generated a dataset with a set of videos where a user is performing a sequence of actions and each image shows the desktop contents at that time point. We also present a framework that is composed of a synthetic sample generation pipeline to augment the dataset with relevant characteristics, and a contrastive learning method to classify images in the videos. We take advantage of the natural conditional, tree-like, relationship of the images' characteristics to regularize the learning of the representations by dealing with multiple partial tasks simultaneously. Experimental results show that the proposed framework outperforms previously proposed hierarchical multi-label contrastive losses in fine-grain UI classification.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 42 of its 59 references that are in the map. First posted on arXiv 15 Mar 2024.

Builds on

42 of 59 references.

References · 42

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  4. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  5. 2020Supervised Contrastive Learningcited by 2,829
  6. 2018Unsupervised Feature Learning via Non-Parametric Instance-level Discriminationcited by 1,845
  7. 2019Contrastive Multiview Codingcited by 1,301
  8. 2018Learning deep representations by mutual information estimation and maximizationcited by 1,273
  9. 2022WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agentscited by 1,251
  10. 2015A Large Dataset to Train Convolutional Networks for Disparity, Optical Flow, and Scene Flow Estimationcited by 881
  11. 2019Data-Efficient Image Recognition with Contrastive Predictive Codingcited by 673
  12. 2022GroupViT: Semantic Segmentation Emerges from Text Supervisioncited by 531
30 more
  1. 2023Language Models can Solve Computer Taskscited by 372
  2. 2019Multi-Similarity Loss with General Pair Weighting for Deep Metric Learningcited by 368
  3. 2014Synthetic Data and Artificial Neural Networks for Natural Scene Text Recognitioncited by 214
  4. 2021DetCo: Unsupervised Contrastive Learning for Object Detectioncited by 205
  5. 2022Unified Contrastive Learning in Image-Text-Label Spacecited by 194
  6. 2019LayoutGAN: Generating Graphic Layouts with Wireframe Discriminatorscited by 131
  7. 2019LayoutVAE: Stochastic Scene Layout Generation From a Label Setcited by 119
  8. 2021UIBert: Learning Generic Multimodal Representations for UI Understandingcited by 95
  9. 2021Variational Transformer Networks for Layout Generationcited by 88
  10. 2022A data-driven approach for learning to control computerscited by 88
  11. 2019Neural Design Network: Graphic Layout Generation with Constraintscited by 83
  12. 2019Sim2real transfer learning for 3D human pose estimation: motion to the rescuecited by 76
  13. 2021BLT: Bidirectional Layout Transformer for Controllable Layout Generationcited by 69
  14. 2023From Pixels to UI Actions: Learning to Follow Instructions via Graphical User Interfacescited by 68
  15. 2022Spotlight: Mobile UI Understanding using Vision-Language Models with a Focuscited by 62
  16. 2022Use All The Labels: A Hierarchical Multi-Label Contrastive Learning Frameworkcited by 59
  17. 2023WebUI: A Dataset for Enhancing Visual UI Understanding with Web Semanticscited by 47
  18. 2020ActionBert: Leveraging User Actions for Semantic Understanding of User Interfacescited by 43
  19. 2021Screen2Vec: Semantic Embedding of GUI Screens and GUI Componentscited by 41
  20. 2021VINS: Visual Search for Mobile User Interface Designcited by 37
  21. 2022Ranking Info Noise Contrastive Estimation: Boosting Contrastive Learning via Ranked Positivescited by 37
  22. 2022Fine-grained Temporal Contrastive Learning for Weakly-supervised Temporal Action Localizationcited by 33
  23. 2020Multi-label Contrastive Predictive Codingcited by 32
  24. 2021VUT: Versatile UI Transformer for Multi-Modal Multi-Task User Interface Modelingcited by 26
  25. 2020Multi-Label Contrastive Learning for Abstract Visual Reasoningcited by 24
  26. 2021Iterative Contrast-Classify For Semi-supervised Temporal Action Segmentationcited by 19
  27. 2021Multi-Label Image Classification with Contrastive Learningcited by 18
  28. 2023High-Resolution Synthetic RGB-D Datasets for Monocular Depth Estimationcited by 10
  29. 2023End-to-End Supervised Multilabel Contrastive Learningcited by 8
  30. 2023SIDAR: Synthetic Image Dataset for Alignment & Restorationcited by 2

Built on by

Not cited yet.