# DFEE: Interactive DataFlow Execution and Evaluation Kit

arXiv: https://arxiv.org/abs/2212.08099 · HTML: https://mlsanity.com/paper/2212.08099/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 4 of its 9 references that are in the map.
- First posted on arXiv 4 Dec 2022.

## Builds on (most cited first, 4 of 4)

- [Task-Oriented Dialogue as Dataflow Synthesis](https://mlsanity.com/paper/2009.11423/) (2020)
- [Learning End-to-End Goal-Oriented Dialog](https://mlsanity.com/paper/1605.07683/) (2016)
- [Low-Resource Domain Adaptation for Compositional Task-Oriented Semantic Parsing](https://mlsanity.com/paper/2010.03546/) (2020)
- [Simplifying Semantic Annotations of SMCalFlow](https://mlsanity.com/paper/2206.13425/) (2022)

## Abstract (arXiv)

DataFlow has been emerging as a new paradigm for building task-oriented chatbots due to its expressive semantic representations of the dialogue tasks. Despite the availability of a large dataset SMCalFlow and a simplified syntax, the development and evaluation of DataFlow-based chatbots remain challenging due to the system complexity and the lack of downstream toolchains. In this demonstration, we present DFEE, an interactive DataFlow Execution and Evaluation toolkit that supports execution, visualization and benchmarking of semantic parsers given dialogue input and backend database. We demonstrate the system via a complex dialog task: event scheduling that involves temporal reasoning. It also supports diagnosing the parsing results via a friendly interface that allows developers to examine dynamic DataFlow and the corresponding execution results. To illustrate how to benchmark SoTA models, we propose a novel benchmark that covers more sophisticated event scheduling scenarios and a new metric on task success evaluation. The codes of DFEE have been released on https://github.com/amazonscience/dataflow-evaluation-toolkit.