# Truth-Conditional Captioning of Time Series Data arXiv: https://arxiv.org/abs/2110.01839 ยท HTML: https://mlsanity.com/paper/2110.01839/ ## Key facts - Cited by 11 papers in the map (data through 14 Oct 2026). - Builds on 33 of its 47 references that are in the map. - First posted on arXiv 5 Oct 2021. ## Builds on (most cited first, 30 of 33) - [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019) - [Neural Discrete Representation Learning](https://mlsanity.com/paper/1711.00937/) (2017) - [DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphs](https://mlsanity.com/paper/1903.00161/) (2019) - [Neural Module Networks](https://mlsanity.com/paper/1511.02799/) (2015) - [FigureQA: An Annotated Figure Dataset for Visual Reasoning](https://mlsanity.com/paper/1710.07300/) (2017) - [Evaluation of Text Generation: A Survey](https://mlsanity.com/paper/2006.14799/) (2020) - [ToTTo: A Controlled Table-To-Text Generation Dataset](https://mlsanity.com/paper/2004.14373/) (2020) - [The E2E Dataset: New Challenges For End-to-End Generation](https://mlsanity.com/paper/1706.09254/) (2017) - [The GEM Benchmark: Natural Language Generation, its Evaluation and Metrics](https://mlsanity.com/paper/2102.01672/) (2021) - [Challenges in Data-to-Document Generation](https://mlsanity.com/paper/1707.08052/) (2017) - [Bottom-Up Abstractive Summarization](https://mlsanity.com/paper/1808.10792/) (2018) - [Faithful to the Original: Fact Aware Neural Abstractive Summarization](https://mlsanity.com/paper/1711.04434/) (2017) - [Annotating and Modeling Fine-grained Factuality in Summarization](https://mlsanity.com/paper/2104.04302/) (2021) - [Handling Divergent Reference Texts when Evaluating Table-to-Text Generation](https://mlsanity.com/paper/1906.01081/) (2019) - [Learning to Describe Differences Between Pairs of Similar Images](https://mlsanity.com/paper/1808.10584/) (2018) - [Logical Natural Language Generation from Open-Domain Tables](https://mlsanity.com/paper/2004.10404/) (2020) - [Learning to Compose Neural Networks for Question Answering](https://mlsanity.com/paper/1601.01705/) (2016) - [Representing Numbers in NLP: a Survey and a Vision](https://mlsanity.com/paper/2103.13136/) (2021) - [Improved Natural Language Generation via Loss Truncation](https://mlsanity.com/paper/2004.14589/) (2020) - [Neural Arithmetic Logic Units](https://mlsanity.com/paper/1808.00508/) (2018) - [Neural Module Networks for Reasoning over Text](https://mlsanity.com/paper/1912.04971/) (2019) - [Text Modular Networks: Learning to Decompose Tasks in the Language of Existing Models](https://mlsanity.com/paper/2009.00751/) (2020) - [Sticking to the Facts: Confident Decoding for Faithful Data-to-Text Generation](https://mlsanity.com/paper/1910.08684/) (2019) - [Giving BERT a Calculator: Finding Operations and Arguments with Reading Comprehension](https://mlsanity.com/paper/1909.00109/) (2019) - [Neuro-Symbolic Visual Reasoning: Disentangling "Visual" from "Reasoning"](https://mlsanity.com/paper/2006.11524/) (2020) - [Unsupervised Discrete Sentence Representation Learning for Interpretable Neural Dialog Generation](https://mlsanity.com/paper/1804.08069/) (2018) - [Towards Faithfulness in Open Domain Table-to-text Generation from an Entity-centric View](https://mlsanity.com/paper/2102.08585/) (2021) - [Learning to Collocate Neural Modules for Image Captioning](https://mlsanity.com/paper/1904.08608/) (2019) - [Figure Captioning with Reasoning and Sequence-Level Training](https://mlsanity.com/paper/1906.02850/) (2019) - [An Empirical Investigation of Contextualized Number Prediction](https://mlsanity.com/paper/2011.07961/) (2020) ## Built on by (most cited first, 11 of 11) - [Language Models Still Struggle to Zero-shot Reason about Time Series](https://mlsanity.com/paper/2404.11757/) (2024) - [LineCap: Line Charts for Data Visualization Captioning Models](https://mlsanity.com/paper/2207.07243/) (2022) - [Empowering Time Series Analysis with Synthetic Data: A Survey and Outlook in the Era of Foundation Models](https://mlsanity.com/paper/2503.11411/) (2025) - [DualSG: A Dual-Stream Explicit Semantic-Guided Multivariate Time Series Forecasting Framework](https://mlsanity.com/paper/2507.21830/) (2025) - [Domain-Independent Automatic Generation of Descriptive Texts for Time-Series Data](https://mlsanity.com/paper/2409.16647/) (2024) - [CLaSP: Learning Concepts for Time-Series Signals from Natural Language Supervision](https://mlsanity.com/paper/2411.08397/) (2024) - [TADACap: Time-series Adaptive Domain-Aware Captioning](https://mlsanity.com/paper/2504.11441/) (2025) - [LETS-C: Leveraging Text Embedding for Time Series Classification](https://mlsanity.com/paper/2407.06533/) (2024) - [KairosVL: Orchestrating Time Series and Semantics for Unified Reasoning](https://mlsanity.com/paper/2602.20494/) (2026) - [Natural Language Interfaces for Spatial and Temporal Databases: A Comprehensive Overview of Methods, Taxonomy, and Future Directions](https://mlsanity.com/paper/2603.23375/) (2026) - [LineupRL: Verifiable Reinforcement Learning for Time Series Captioning via Caption-to-Series Identification](https://mlsanity.com/paper/2610.01800/) (2026) ## Abstract (arXiv) In this paper, we explore the task of automatically generating natural language descriptions of salient patterns in a time series, such as stock prices of a company over a week. A model for this task should be able to extract high-level patterns such as presence of a peak or a dip. While typical contemporary neural models with attention mechanisms can generate fluent output descriptions for this task, they often generate factually incorrect descriptions. We propose a computational model with a truth-conditional architecture which first runs small learned programs on the input time series, then identifies the programs/patterns which hold true for the given input, and finally conditions on only the chosen valid program (rather than the input time series) to generate the output text description. A program in our model is constructed from modules, which are small neural networks that are designed to capture numerical patterns and temporal information. The modules are shared across multiple programs, enabling compositionality as well as efficient learning of module parameters. The modules, as well as the composition of the modules, are unobserved in data, and we learn them in an end-to-end fashion with the only training signal coming from the accompanying natural language text descriptions. We find that the proposed model is able to generate high-precision captions even though we consider a small and simple space of module types.