# Compositional Benchmark Synthesis for Hierarchical Human Action Recognition

arXiv: https://arxiv.org/abs/2608.10765 · HTML: https://mlsanity.com/paper/2608.10765/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 24 references that are in the map.
- First posted on arXiv 11 Aug 2026.

## Builds on (most cited first, 18 of 18)

- [Datasheets for Datasets](https://mlsanity.com/paper/1803.09010/) (2018)
- [Heterogeneous Graph Transformer](https://mlsanity.com/paper/2003.01332/) (2020)
- [NTU RGB+D 120: A Large-Scale Benchmark for 3D Human Activity Understanding](https://mlsanity.com/paper/1905.04757/) (2019)
- [Assembly101: A Large-Scale Multi-View Video Dataset for Understanding Procedural Activities](https://mlsanity.com/paper/2203.14712/) (2022)
- [A Semantic Loss Function for Deep Learning with Symbolic Knowledge](https://mlsanity.com/paper/1711.11157/) (2017)
- [COGS: A Compositional Generalization Challenge Based on Semantic Interpretation](https://mlsanity.com/paper/2010.05465/) (2020)
- [Something-Else: Compositional Action Recognition with Spatial-Temporal Interaction Networks](https://mlsanity.com/paper/1912.09930/) (2019)
- [Intention-Conditioned Long-Term Human Egocentric Action Forecasting](https://mlsanity.com/paper/2207.12080/) (2022)
- [Procedural Generation of Videos to Train Deep Action Recognition Networks](https://mlsanity.com/paper/1612.00881/) (2016)
- [Predicting Human Activities Using Stochastic Grammar](https://mlsanity.com/paper/1708.00945/) (2017)
- [Adversarial Generative Grammars for Human Activity Prediction](https://mlsanity.com/paper/2008.04888/) (2020)
- [C2C: Component-to-Composition Learning for Zero-Shot Compositional Action Recognition](https://mlsanity.com/paper/2407.06113/) (2024)
- [What Has Been Lost with Synthetic Evaluation?](https://mlsanity.com/paper/2505.22830/) (2025)
- [Gaze-Guided Graph Neural Network for Action Anticipation Conditioned on Intention](https://mlsanity.com/paper/2404.07347/) (2024)
- [Vision and Intention Boost Large Language Model in Long-Term Action Anticipation](https://mlsanity.com/paper/2505.01713/) (2025)
- [Enhancing Action Recognition by Leveraging the Hierarchical Structure of Actions and Textual Context](https://mlsanity.com/paper/2410.21275/) (2024)
- [Inferring Temporal Compositions of Actions Using Probabilistic Automata](https://mlsanity.com/paper/2004.13217/) (2020)
- [LogicMP: A Neuro-symbolic Approach for Encoding First-order Logic Constraints](https://mlsanity.com/paper/2309.15458/) (2023)

## Built on by (most cited first, 1 of 1)

- [Neuro-Symbolic Graph Transformers for Hierarchical Intention Recognition from Multimodal Human Activity Sequences](https://mlsanity.com/paper/2610.18176/) (2026)

## Abstract (arXiv)

Recognizing human behavior across levels of abstraction, from atomic actions to long-horizon intentions, requires data annotated along a semantic hierarchy. Large corpora provide isolated, atomically labeled clips without temporal composition, whereas recorded composite-activity corpora offer shallow, domain-narrow, fixedhierarchies. A benchmark-generation and evaluation frameworkis proposed that synthesizes a four-level hierarchical-intention benchmark, spanning actions, activities, low-level intentions (LLIs), and high-level intentions (HLIs), from a flat single-label action corpus while retaining real pre-extracted features at the action level. Episodes are assembled by a transition model under a subject-consistency constraint, and a coverage-aware sampler reduces the subject usage Gini from 0.566 to 0.248. Synthesizing such a benchmark raises a circular-supervision risk that recorded datasets avoid: if the rules generating the episodes also govern the evaluation, models can succeed by recovering the generator rather than through genuine reasoning. Validity is addressed by design, holding sequence-generation rules disjoint from the first-order-logic rules used at evaluation. The instantiation yields 15,002 episodes. Four reference baselines from different model families characterize difficulty, not as recognition methods. A compositional held-out gap of 0.13 to 0.17 macro-F1 appears across all baselines, including a graph-aware model that recognizes best yet does not close the gap, indicating a structural property of the benchmark rather than a model artifact. A logic-free baseline still violates the held-out semantic rules above their intrinsic data rate, and the order-destroying control changes macro-F1 within seed variation, serving as a generator-consistency check. Theontology, transition model, and generator are released so the benchmark can beregenerated and extended.