# TagRuler: Interactive Tool for Span-Level Data Programming by Demonstration

arXiv: https://arxiv.org/abs/2106.12767 · HTML: https://mlsanity.com/paper/2106.12767/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 10 of its 24 references that are in the map.
- First posted on arXiv 24 Jun 2021.

## Builds on (most cited first, 10 of 10)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Deep contextualized word representations](https://mlsanity.com/paper/1802.05365/) (2018)
- [Snorkel: Rapid Training Data Creation with Weak Supervision](https://mlsanity.com/paper/1711.10160/) (2017)
- [Dice Loss for Data-imbalanced NLP Tasks](https://mlsanity.com/paper/1911.02855/) (2019)
- [Fast and Three-rious: Speeding Up Weak Supervision with Triplet Methods](https://mlsanity.com/paper/2002.11955/) (2020)
- [Training Classifiers with Natural Language Explanations](https://mlsanity.com/paper/1805.03818/) (2018)
- [Interactive Weak Supervision: Learning Useful Heuristics for Data Labeling](https://mlsanity.com/paper/2012.06046/) (2020)
- [Interactive Task and Concept Learning from Natural Language Instructions and GUI Demonstrations](https://mlsanity.com/paper/1909.00031/) (2019)
- [Leveraging Organizational Resources to Adapt Models to New Data Modalities](https://mlsanity.com/paper/2008.09983/) (2020)

## Built on by (most cited first, 4 of 4)

- [PRBoost: Prompt-Based Rule Discovery and Boosting for Interactive Weakly-Supervised Learning](https://mlsanity.com/paper/2203.09735/) (2022)
- [Human-in-the-Loop Synthetic Text Data Inspection with Provenance Tracking](https://mlsanity.com/paper/2404.18881/) (2024)
- [Adaptive Multi-view Rule Discovery for Weakly-Supervised Compatible Products Prediction](https://mlsanity.com/paper/2206.13749/) (2022)
- [ProTAL: A Drag-and-Link Video Programming Framework for Temporal Action Localization](https://mlsanity.com/paper/2505.17555/) (2025)

## Abstract (arXiv)

Despite rapid developments in the field of machine learning research, collecting high-quality labels for supervised learning remains a bottleneck for many applications. This difficulty is exacerbated by the fact that state-of-the-art models for NLP tasks are becoming deeper and more complex, often increasing the amount of training data required even for fine-tuning. Weak supervision methods, including data programming, address this problem and reduce the cost of label collection by using noisy label sources for supervision. However, until recently, data programming was only accessible to users who knew how to program. To bridge this gap, the Data Programming by Demonstration framework was proposed to facilitate the automatic creation of labeling functions based on a few examples labeled by a domain expert. This framework has proven successful for generating high-accuracy labeling models for document classification. In this work, we extend the DPBD framework to span-level annotation tasks, arguably one of the most time-consuming NLP labeling tasks. We built a novel tool, TagRuler, that makes it easy for annotators to build span-level labeling functions without programming and encourages them to explore trade-offs between different labeling models and active learning strategies. We empirically demonstrated that an annotator could achieve a higher F1 score using the proposed tool compared to manual labeling for different span-level annotation tasks.