# End-to-End Fine-Grained Action Segmentation and Recognition Using Conditional Random Field Models and Discriminative Sparse Coding

arXiv: https://arxiv.org/abs/1801.09571 · HTML: https://mlsanity.com/paper/1801.09571/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 29 Jan 2018.

## Built on by (most cited first, 9 of 9)

- [Gesture Recognition in Robotic Surgery: a Review](https://mlsanity.com/paper/2102.00027/) (2021)
- [Tubelet-Contrastive Self-Supervision for Video-Efficient Generalization](https://mlsanity.com/paper/2303.11003/) (2023)
- [Weakly-Supervised Temporal Action Detection for Fine-Grained Videos with Hierarchical Atomic Actions](https://mlsanity.com/paper/2207.11805/) (2022)
- [Surgical Gesture Recognition Based on Bidirectional Multi-Layer Independently RNN with Explainable Spatial Feature Extraction](https://mlsanity.com/paper/2105.00460/) (2021)
- [LocoMotion: Learning Motion-Focused Video-Language Representations](https://mlsanity.com/paper/2410.12018/) (2024)
- [MS-TCRNet: Multi-Stage Temporal Convolutional Recurrent Networks for Action Segmentation Using Sensor-Augmented Kinematics](https://mlsanity.com/paper/2303.07814/) (2023)
- [Dual-Stream Alignment for Action Segmentation](https://mlsanity.com/paper/2510.07652/) (2025)
- [Facial Tic Detection in Untrimmed Videos of Tourette Syndrome Patients](https://mlsanity.com/paper/2211.03895/) (2022)
- [Improving Temporal Action Segmentation via Constraint-Aware Decoding](https://mlsanity.com/paper/2605.10149/) (2026)

## Abstract (arXiv)

Fine-grained action segmentation and recognition is an important yet challenging task. Given a long, untrimmed sequence of kinematic data, the task is to classify the action at each time frame and segment the time series into the correct sequence of actions. In this paper, we propose a novel framework that combines a temporal Conditional Random Field (CRF) model with a powerful frame-level representation based on discriminative sparse coding. We introduce an end-to-end algorithm for jointly learning the weights of the CRF model, which include action classification and action transition costs, as well as an overcomplete dictionary of mid-level action primitives. This results in a CRF model that is driven by sparse coding features obtained using a discriminative dictionary that is shared among different actions and adapted to the task of structured output learning. We evaluate our method on three surgical tasks using kinematic data from the JIGSAWS dataset, as well as on a food preparation task using accelerometer data from the 50 Salads dataset. Our results show that the proposed method performs on par or better than state-of-the-art methods.