# CASR: Refining Action Segmentation via Marginalizing Frame-levle Causal Relationships

arXiv: https://arxiv.org/abs/2311.12401 · HTML: https://mlsanity.com/paper/2311.12401/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 36 references that are in the map.
- First posted on arXiv 21 Nov 2023.

## Builds on (most cited first, 20 of 20)

- [Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset](https://mlsanity.com/paper/1705.07750/) (2017)
- [CLEVRER: CoLlision Events for Video REpresentation and Reasoning](https://mlsanity.com/paper/1910.01442/) (2019)
- [MS-TCN: Multi-Stage Temporal Convolutional Network for Action Segmentation](https://mlsanity.com/paper/1903.01945/) (2019)
- [Whitening for Self-Supervised Representation Learning](https://mlsanity.com/paper/2007.06346/) (2020)
- [ASFormer: Transformer for Action Segmentation](https://mlsanity.com/paper/2110.08568/) (2021)
- [Causal Consistency of Structural Equation Models](https://mlsanity.com/paper/1707.00819/) (2017)
- [Abstracting Causal Models](https://mlsanity.com/paper/1812.03789/) (2018)
- [Alleviating Over-segmentation Errors by Detecting Action Boundaries](https://mlsanity.com/paper/2007.06866/) (2020)
- [Deconfounded Video Moment Retrieval with Causal Intervention](https://mlsanity.com/paper/2106.01534/) (2021)
- [Action Segmentation with Joint Self-Supervised Temporal Domain Adaptation](https://mlsanity.com/paper/2003.02824/) (2020)
- [Unified Fully and Timestamp Supervised Temporal Action Segmentation via Sequence to Sequence Translation](https://mlsanity.com/paper/2209.00638/) (2022)
- [Compositional Abstraction Error and a Category of Causal Models](https://mlsanity.com/paper/2103.15758/) (2021)
- [Temporal Relational Modeling with Self-Supervision for Action Segmentation](https://mlsanity.com/paper/2012.07508/) (2020)
- [Coarse to Fine Multi-Resolution Temporal Convolutional Network](https://mlsanity.com/paper/2105.10859/) (2021)
- [PC-HMR: Pose Calibration for 3D Human Mesh Recovery from 2D Images/Videos](https://mlsanity.com/paper/2103.09009/) (2021)
- [Causal Deep Learning](https://mlsanity.com/paper/2303.02186/) (2023)
- [Cross-Enhancement Transformer for Action Segmentation](https://mlsanity.com/paper/2205.09445/) (2022)
- [On the Equivalence of Causal Models: A Category-Theoretic Approach](https://mlsanity.com/paper/2201.06981/) (2022)
- [SOAR: Scene-debiasing Open-set Action Recognition](https://mlsanity.com/paper/2309.01265/) (2023)
- [Deep Causal Learning: Representation, Discovery and Inference](https://mlsanity.com/paper/2211.03374/) (2022)

## Built on by (most cited first, 1 of 1)

- [Towards Causal Relationship in Indefinite Data: Baseline Model and New Datasets](https://mlsanity.com/paper/2401.08221/) (2024)

## Abstract (arXiv)

Integrating deep learning and causal discovery has increased the interpretability of Temporal Action Segmentation (TAS) tasks. However, frame-level causal relationships exist many complicated noises outside the segment-level, making it infeasible to directly express macro action semantics. Thus, we propose Causal Abstraction Segmentation Refiner (CASR), which can refine TAS results from various models by enhancing video causality in marginalizing frame-level casual relationships. Specifically, we define the equivalent frame-level casual model and segment-level causal model, so that the causal adjacency matrix constructed from marginalized frame-level causal relationships has the ability to represent the segmnet-level causal relationships. CASR works out by reducing the difference in the causal adjacency matrix between we constructed and pre-segmentation results of backbone models. In addition, we propose a novel evaluation metric Causal Edit Distance (CED) to evaluate the causal interpretability. Extensive experimental results on mainstream datasets indicate that CASR significantly surpasses existing various methods in action segmentation performance, as well as in causal explainability and generalization.