Responsible Task Automation: Empowering Large Language Models as Responsible Task Automators
Not on the map. Why?
Abstract (arXiv)
The recent success of Large Language Models (LLMs) signifies an impressive stride towards artificial general intelligence. They have shown a promising prospect in automatically completing tasks upon user instructions, functioning as brain-like coordinators. The associated risks will be revealed as we delegate an increasing number of tasks to machines for automated completion. A big question emerges: how can we make machines behave responsibly when helping humans automate tasks as personal copilots? In this paper, we explore this question in depth from the perspectives of feasibility, completeness and security. In specific, we present Responsible Task Automation (ResponsibleTA) as a fundamental framework to facilitate responsible collaboration between LLM-based coordinators and executors for task automation with three empowered capabilities: 1) predicting the feasibility of the commands for executors; 2) verifying the completeness of executors; 3) enhancing the security (e.g., the protection of users' privacy). We further propose and compare two paradigms for implementing the first two capabilities. One is to leverage the generic knowledge of LLMs themselves via prompt engineering while the other is to adopt domain-specific learnable models. Moreover, we introduce a local memory mechanism for achieving the third capability. We evaluate our proposed ResponsibleTA on UI task automation and hope it could bring more attentions to ensuring LLMs more responsible in diverse scenarios.
Key facts. Cited by 14 papers in the map (data through 7 Oct 2026). Builds on 32 of its 46 references that are in the map. First posted on arXiv 2 Jun 2023.
Builds on
32 of 46 references.
References · 32
- 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
- 2014Adam: A Method for Stochastic Optimizationcited by 36,960
- 2020Language Models are Few-Shot Learnerscited by 32,199
- 2022Training language models to follow instructions with human feedbackcited by 15,712
- 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
- 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
- 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
- 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
- 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
- 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
- 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,302
- 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,037
20 more
- 2023HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Facecited by 1,173
- 2020ALFWorld: Aligning Text and Embodied Environments for Interactive Learningcited by 1,143
- 2018ShuffleNet V2: Practical Guidelines for Efficient CNN Architecture Designcited by 1,132
- 2022Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agentscited by 1,029
- 2023Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Modelscited by 659
- 2023Language Is Not All You Need: Aligning Perception with Language Modelscited by 543
- 2023MM-REACT: Prompting ChatGPT for Multimodal Reasoning and Actioncited by 511
- 2016The Option-Critic Architecturecited by 433
- 2021OCR-free Document Understanding Transformercited by 400
- 2023Describe, Explain, Plan and Select: Interactive Planning with Large Language Models Enables Open-World Multi-Task Agentscited by 396
- 2021Pix2seq: A Language Modeling Framework for Object Detectioncited by 345
- 2023ChatGPT for Robotics: Design Principles and Model Abilitiescited by 296
- 2022RTMDet: An Empirical Study of Designing Real-Time Object Detectorscited by 192
- 2023TaskMatrix.AI: Completing Tasks by Connecting Foundation Models with Millions of APIscited by 184
- 2022A Unified Sequence Interface for Vision Taskscited by 152
- 2021UniTAB: Unifying Text and Box Outputs for Grounded Vision-Language Modelingcited by 122
- 2022PP-OCRv3: More Attempts for the Improvement of Ultra Lightweight OCR Systemcited by 105
- 2022On Grounded Planning for Embodied Tasks with Language Modelscited by 33
- 2023Plan, Eliminate, and Track -- Language Models are Good Teachers for Embodied Agentscited by 32
- 2021Language Models are Few-Shot Butlerscited by 27
Built on by
Citing papers per year: 2023: 2; 2024: 7; 2025: 3; 2026: 2.
14 papers.
Sort:
By area:
- 2024Personal LLM Agents: Insights and Survey about the Capability, Efficiency and Securitycited by 257
- 2023AutoDroid: LLM-powered Task Automation in Androidcited by 178
- 2025Deep Video Discovery: Agentic Search with Tool Use for Long-form Video Understandingcited by 94
- 2023AXNav: Replaying Accessibility Tests from Natural Languagecited by 31
- 2024AutoDroid-V2: Boosting SLM-based GUI Agents via Code Generationcited by 31
- 2024VisualCritic: Making LMMs Perceive Visual Quality Like Humanscited by 13
- 2024From Interaction to Impact: Towards Safer AI Agents Through Understanding and Evaluating Mobile UI Operation Impactscited by 10
- 2024Steward: Natural Language Web Automationcited by 4
- 2024Text Grouping Adapter: Adapting Pre-trained Text Detector for Layout Analysiscited by 2
- 2025UI-Evol: Automatic Knowledge Evolving for Computer Use Agentscited by 1
- 2025Synthetic Cognitive Walkthrough: Aligning Large Language Model Performance with Human Cognitive Walkthroughcited by 1
- 2026GhostUI: Unveiling Hidden Interactions in Mobile UIcited by 1
- 2026From Content to Knowledge: Lightning Fast Long-Video Understanding with Neural Knowledge Representations10 Jun 2026
- 2026GhostUI: Unveiling Hidden Interactions in Mobile UI27 Jan 2026
- 2025Synthetic Cognitive Walkthrough: Aligning Large Language Model Performance with Human Cognitive Walkthrough3 Dec 2025
- 2025UI-Evol: Automatic Knowledge Evolving for Computer Use Agents28 May 2025
- 2025Deep Video Discovery: Agentic Search with Tool Use for Long-form Video Understanding23 May 2025
- 2024AutoDroid-V2: Boosting SLM-based GUI Agents via Code Generation24 Dec 2024
- 2024From Interaction to Impact: Towards Safer AI Agents Through Understanding and Evaluating Mobile UI Operation Impacts11 Oct 2024
- 2024Steward: Natural Language Web Automation23 Sep 2024
- 2024Class-Conditional self-reward mechanism for improved Text-to-Image models22 May 2024
- 2024Text Grouping Adapter: Adapting Pre-trained Text Detector for Layout Analysis13 May 2024
- 2024VisualCritic: Making LMMs Perceive Visual Quality Like Humans19 Mar 2024
- 2024Personal LLM Agents: Insights and Survey about the Capability, Efficiency and Security10 Jan 2024
2 more
- 2023AXNav: Replaying Accessibility Tests from Natural Language3 Oct 2023
- 2023AutoDroid: LLM-powered Task Automation in Android29 Aug 2023
- 2024Personal LLM Agents: Insights and Survey about the Capability, Efficiency and Securitycited by 257
- 2025Deep Video Discovery: Agentic Search with Tool Use for Long-form Video Understandingcited by 94
- 2024AutoDroid-V2: Boosting SLM-based GUI Agents via Code Generationcited by 31
- 2024From Interaction to Impact: Towards Safer AI Agents Through Understanding and Evaluating Mobile UI Operation Impactscited by 10
- 2025UI-Evol: Automatic Knowledge Evolving for Computer Use Agentscited by 1
- 2025UI-Evol: Automatic Knowledge Evolving for Computer Use Agents28 May 2025
- 2025Deep Video Discovery: Agentic Search with Tool Use for Long-form Video Understanding23 May 2025
- 2024AutoDroid-V2: Boosting SLM-based GUI Agents via Code Generation24 Dec 2024
- 2024From Interaction to Impact: Towards Safer AI Agents Through Understanding and Evaluating Mobile UI Operation Impacts11 Oct 2024
- 2024Personal LLM Agents: Insights and Survey about the Capability, Efficiency and Security10 Jan 2024
- 2023AutoDroid: LLM-powered Task Automation in Androidcited by 178
- 2023AXNav: Replaying Accessibility Tests from Natural Languagecited by 31
- 2024Steward: Natural Language Web Automationcited by 4
- 2024Text Grouping Adapter: Adapting Pre-trained Text Detector for Layout Analysiscited by 2
- 2026GhostUI: Unveiling Hidden Interactions in Mobile UIcited by 1
- 2026From Content to Knowledge: Lightning Fast Long-Video Understanding with Neural Knowledge Representationscited by 0
- 2026From Content to Knowledge: Lightning Fast Long-Video Understanding with Neural Knowledge Representations10 Jun 2026
- 2026GhostUI: Unveiling Hidden Interactions in Mobile UI27 Jan 2026
- 2024Steward: Natural Language Web Automation23 Sep 2024
- 2024Text Grouping Adapter: Adapting Pre-trained Text Detector for Layout Analysis13 May 2024
- 2023AXNav: Replaying Accessibility Tests from Natural Language3 Oct 2023
- 2023AutoDroid: LLM-powered Task Automation in Android29 Aug 2023