# TV-TREES: Multimodal Entailment Trees for Neuro-Symbolic Video Reasoning

arXiv: https://arxiv.org/abs/2402.19467 · HTML: https://mlsanity.com/paper/2402.19467/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Placed in Multimodal reasoning, one of 17 directions in LLM reasoning.
- Builds on 21 of its 53 references that are in the map.
- First posted on arXiv 29 Feb 2024.

## Placed in

- LLM reasoning / [Multimodal reasoning](https://mlsanity.com/lr/multimodal-reasoning/)

## Builds on (most cited first, 21 of 21)

- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference](https://mlsanity.com/paper/1704.05426/) (2017)
- [A large annotated corpus for learning natural language inference](https://mlsanity.com/paper/1508.05326/) (2015)
- [MS MARCO: A Human Generated MAchine Reading COmprehension Dataset](https://mlsanity.com/paper/1611.09268/) (2016)
- [MVBench: A Comprehensive Multi-modal Video Understanding Benchmark](https://mlsanity.com/paper/2311.17005/) (2023)
- [TVQA: Localized, Compositional Video Question Answering](https://mlsanity.com/paper/1809.01696/) (2018)
- [HERO: Hierarchical Encoder for Video+Language Omni-representation Pre-training](https://mlsanity.com/paper/2005.00200/) (2020)
- [Evaluation of Text Generation: A Survey](https://mlsanity.com/paper/2006.14799/) (2020)
- [Visual Entailment: A Novel Task for Fine-Grained Image Understanding](https://mlsanity.com/paper/1901.06706/) (2019)
- [Self-Chained Image-Language Model for Video Localization and Question Answering](https://mlsanity.com/paper/2305.06988/) (2023)
- [TVQA+: Spatio-Temporal Grounding for Video Question Answering](https://mlsanity.com/paper/1904.11574/) (2019)
- [Video Question Answering: Datasets, Algorithms and Challenges](https://mlsanity.com/paper/2203.01225/) (2022)
- [Understanding by Understanding Not: Modeling Negation in Language Models](https://mlsanity.com/paper/2105.03519/) (2021)
- [Large Language Models are Temporal and Causal Reasoners for Video Question Answering](https://mlsanity.com/paper/2310.15747/) (2023)
- [e-SNLI-VE: Corrected Visual-Textual Entailment with Natural Language Explanations](https://mlsanity.com/paper/2004.03744/) (2020)
- [Distilled Dual-Encoder Model for Vision-Language Understanding](https://mlsanity.com/paper/2112.08723/) (2021)
- [Enhancing Systematic Decompositional Natural Language Inference Using Informal Logic](https://mlsanity.com/paper/2402.14798/) (2024)
- [Multimodal Logical Inference System for Visual-Textual Entailment](https://mlsanity.com/paper/1906.03952/) (2019)
- [Don't Explain without Verifying Veracity: An Evaluation of Explainable AI with Video Activity Recognition](https://mlsanity.com/paper/2005.02335/) (2020)
- [Dissecting Multimodality in VideoQA Transformer Models by Impairing Modality Fusion](https://mlsanity.com/paper/2306.08889/) (2023)
- [Explainable Deep Learning for Video Recognition Tasks: A Framework & Recommendations](https://mlsanity.com/paper/1909.05667/) (2019)

## Built on by (most cited first, 12 of 12)

- [VideoTree: Adaptive Tree-based Video Representation for LLM Reasoning on Long Videos](https://mlsanity.com/paper/2405.19209/) (2024)
- [Explainable and Interpretable Multimodal Large Language Models: A Comprehensive Survey](https://mlsanity.com/paper/2412.02104/) (2024)
- [Neuro-Symbolic AI in 2024: A Systematic Review](https://mlsanity.com/paper/2501.05435/) (2025)
- [Commonsense Video Question Answering through Video-Grounded Entailment Tree Reasoning](https://mlsanity.com/paper/2501.05069/) (2025)
- [Natural Language Inference Improves Compositionality in Vision-Language Models](https://mlsanity.com/paper/2410.22315/) (2024)
- [LVC: A Lightweight Compression Framework for Enhancing VLMs in Long Video Understanding](https://mlsanity.com/paper/2504.06835/) (2025)
- [A Study on Neuro-Symbolic Artificial Intelligence: Healthcare Perspectives](https://mlsanity.com/paper/2503.18213/) (2025)
- [Gaps or Hallucinations? Gazing into Machine-Generated Legal Analysis for Fine-grained Text Evaluations](https://mlsanity.com/paper/2409.09947/) (2024)
- [MDSEval: A Meta-Evaluation Benchmark for Multimodal Dialogue Summarization](https://mlsanity.com/paper/2510.01659/) (2025)
- [MMViR: A Multi-Modal and Multi-Granularity Representation for Long-range Video Understanding](https://mlsanity.com/paper/2601.05495/) (2026)
- [Bonsai: Interpretable Tree-Adaptive Grounded Reasoning](https://mlsanity.com/paper/2504.03640/) (2025)
- [VideoForest: Person-Anchored Hierarchical Reasoning for Cross-Video Question Answering](https://mlsanity.com/paper/2508.03039/) (2025)

## Abstract (arXiv)

It is challenging for models to understand complex, multimodal content such as television clips, and this is in part because video-language models often rely on single-modality reasoning and lack interpretability. To combat these issues we propose TV-TREES, the first multimodal entailment tree generator. TV-TREES serves as an approach to video understanding that promotes interpretable joint-modality reasoning by searching for trees of entailment relationships between simple text-video evidence and higher-level conclusions that prove question-answer pairs. We also introduce the task of multimodal entailment tree generation to evaluate reasoning quality. Our method's performance on the challenging TVQA benchmark demonstrates interpretable, state-of-the-art zero-shot performance on full clips, illustrating that multimodal entailment tree generation can be a best-of-both-worlds alternative to black-box systems.