mlsanityData to 14 Oct 2026

TWIZ-v2: The Wizard of Multimodal Conversational-Stimulus

arXiv 2310.02118 ↗First posted 3 Oct 2023cs.CL cs.AIcited by 2

Not on the map. Why?

Abstract (arXiv)

In this report, we describe the vision, challenges, and scientific contributions of the Task Wizard team, TWIZ, in the Alexa Prize TaskBot Challenge 2022. Our vision, is to build TWIZ bot as an helpful, multimodal, knowledgeable, and engaging assistant that can guide users towards the successful completion of complex manual tasks. To achieve this, we focus our efforts on three main research questions: (1) Humanly-Shaped Conversations, by providing information in a knowledgeable way; (2) Multimodal Stimulus, making use of various modalities including voice, images, and videos; and (3) Zero-shot Conversational Flows, to improve the robustness of the interaction to unseen scenarios. TWIZ is an assistant capable of supporting a wide range of tasks, with several innovative features such as creative cooking, video navigation through voice, and the robust TWIZ-LLM, a Large Language Model trained for dialoguing about complex manual tasks. Given ratings and feedback provided by users, we observed that TWIZ bot is an effective and robust system, capable of guiding users through tasks while providing several multimodal stimuli.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 31 of its 44 references that are in the map. First posted on arXiv 3 Oct 2023.

Builds on

31 of 44 references.

References · 31

  1. 2017Attention Is All You Needcited by 60,361
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  4. 2022Training language models to follow instructions with human feedbackcited by 15,971
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  6. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  7. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  8. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,512
  9. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,356
  10. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,322
  11. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  12. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
19 more
  1. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,876
  2. 2020Learning to summarize from human feedbackcited by 2,588
  3. 2022Survey of Hallucination in Natural Language Generationcited by 2,075
  4. 2020MPNet: Masked and Permuted Pre-training for Language Understandingcited by 981
  5. 2019Adversarial NLI: A New Benchmark for Natural Language Understandingcited by 807
  6. 2022OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Frameworkcited by 745
  7. 2022GIT: A Generative Image-to-text Transformer for Vision and Languagecited by 537
  8. 2020UnifiedQA: Crossing Format Boundaries With a Single QA Systemcited by 476
  9. 2023TIFA: Accurate and Interpretable Text-to-Image Faithfulness Evaluation with Question Answeringcited by 428
  10. 2022mPLUG: Effective and Efficient Vision-Language Learning by Cross-modal Skip-connectionscited by 187
  11. 2021Q2: Evaluating Factual Consistency in Knowledge-Grounded Dialogues via Question Generation and Question Answeringcited by 132
  12. 2020Language Model is All You Need: Natural Language Understanding as Question Answeringcited by 29
  13. 2022Alexa, Let's Work Together: Introducing the First Alexa Prize TaskBot Challenge on Conversational Task Assistancecited by 15
  14. 2020Offline and Online Satisfaction Prediction in Open-Domain Conversational Systemscited by 12
  15. 2023Improving Open-Domain Dialogue Evaluation with a Causal Inference Modelcited by 4
  16. 2023Transferring Visual Attributes from Natural Language to Verified Image Generationcited by 4
  17. 2023Task Conditioned BERT for Joint Intent Detection and Slot-fillingcited by 3
  18. 2023The Wizard of Curiosities: Enriching Dialogues with Fun Factscited by 2
  19. 2023Rating Prediction in Conversational Task Assistants with Behavioral and Conversational-Flow Featurescited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 0; 2026: 1.

2 papers.

  1. 2024Gl'orIA -- A Generative and Open Large Language Model for Portuguesecited by 16
  2. 2026Large-Scale User Behavior Analysis in Multimodal AI-Assisted Manual Task Executioncited by 0