mlsanityData to 14 Oct 2026

CLMSM: A Multi-Task Learning Framework for Pre-training on Procedural Text

arXiv 2310.14326 ↗First posted 22 Oct 2023cs.CL cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

In this paper, we propose CLMSM, a domain-specific, continual pre-training framework, that learns from a large set of procedural recipes. CLMSM uses a Multi-Task Learning Framework to optimize two objectives - a) Contrastive Learning using hard triplets to learn fine-grained differences across entities in the procedures, and b) a novel Mask-Step Modelling objective to learn step-wise context of a procedure. We test the performance of CLMSM on the downstream tasks of tracking entities and aligning actions between two procedures on three datasets, one of which is an open-domain dataset not conforming with the pre-training dataset. We show that CLMSM not only outperforms baselines on recipes (in-domain) but is also able to generalize to open-domain procedural NLP tasks.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 23 of its 36 references that are in the map. First posted on arXiv 22 Oct 2023.

Builds on

23 of 36 references.

References · 23

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  4. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  5. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,893
  6. 2018Know What You Don't Know: Unanswerable Questions for SQuADcited by 1,554
  7. 2023The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Onlycited by 763
  8. 2020SPECTER: Document-level Representation Learning using Citation-informed Transformerscited by 430
  9. 2018Recipe1M+: A Dataset for Learning Cross-Modal Embeddings for Cooking Recipes and Food Imagescited by 118
  10. 2022Neighborhood Contrastive Learning for Scientific Document Representations with Citation Embeddingscited by 75
  11. 2018Tracking State Changes in Procedural Text: A Challenge Dataset and Models for Process Paragraph Comprehensioncited by 62
  12. 2017Simulating Action Dynamics with Neural Process Networkscited by 53
11 more
  1. 2019Generating Personalized Recipes from Historical User Preferencescited by 53
  2. 2016Tracking the World State with Recurrent Entity Networkscited by 51
  3. 2019Everything Happens for a Reason: Discovering the Purpose of Actions in Procedural Textcited by 31
  4. 2023Learning and Verification of Task Structure in Instructional Videoscited by 26
  5. 2021Time-Stamped Language Model: Teaching Language Models to Understand the Flow of Eventscited by 22
  6. 2021Question Answering over Electronic Devices: A New Benchmark Dataset and a Multi-Task Learning based QA Frameworkcited by 21
  7. 2018Building Dynamic Knowledge Graphs from Text using Machine Reading Comprehensioncited by 17
  8. 2020Procedural Reading Comprehension with Attribute-Aware Context Flowcited by 16
  9. 2019Tracking Discrete and Continuous Entity State for Process Understandingcited by 15
  10. 2020A Recipe for Creating Multimodal Aligned Datasets for Sequential Taskscited by 15
  11. 2018What Happened? Leveraging VerbNet to Predict the Effects of Actions in Procedural Textcited by 5

Built on by

Not cited yet.