# ByteAction: Byte-space Action Recognition Foundation Model

arXiv: https://arxiv.org/abs/2608.22760 · HTML: https://mlsanity.com/paper/2608.22760/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 16 of its 36 references that are in the map.
- First posted on arXiv 24 Aug 2026.

## Builds on (most cited first, 16 of 16)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks](https://mlsanity.com/paper/2312.14238/) (2023)
- [Detecting and Recognizing Human-Object Interactions](https://mlsanity.com/paper/1704.07333/) (2017)
- [Byte Latent Transformer: Patches Scale Better Than Tokens](https://mlsanity.com/paper/2412.09871/) (2024)
- [Contextual Action Recognition with R*CNN](https://mlsanity.com/paper/1505.01197/) (2015)
- [Detecting Any Human-Object Interaction Relationship: Universal HOI Detector with Spatial Prompt Learning on Foundation Models](https://mlsanity.com/paper/2311.03799/) (2023)
- [MrT5: Dynamic Token Merging for Efficient Byte-level Language Models](https://mlsanity.com/paper/2410.20771/) (2024)
- [Beyond Language Models: Byte Models are Digital World Simulators](https://mlsanity.com/paper/2402.19155/) (2024)
- [Bytes Are All You Need: Transformers Operating Directly On File Bytes](https://mlsanity.com/paper/2306.00238/) (2023)
- [Skeleton-based Gesture Recognition Using Several Fully Connected Layers with Path Signature Features and Temporal Transformer Module](https://mlsanity.com/paper/1811.07081/) (2018)
- [Representation Potentials of Foundation Models for Multimodal Alignment: A Survey](https://mlsanity.com/paper/2510.05184/) (2025)
- [Bitstream-Corrupted JPEG Images are Restorable: Two-stage Compensation and Alignment Framework for Image Restoration](https://mlsanity.com/paper/2304.06976/) (2023)
- [ByteNet: Rethinking Multimedia File Fragment Classification through Visual Perspectives](https://mlsanity.com/paper/2410.20855/) (2024)
- [Transform and Bitstream Domain Image Classification](https://mlsanity.com/paper/2110.06740/) (2021)
- [Human Action Recognition in Still Images Using ConViT](https://mlsanity.com/paper/2307.08994/) (2023)
- [Bitstream Action Recognition is Byte Modeling](https://mlsanity.com/paper/2608.15695/) (2026)

## Abstract (arXiv)

Byte-space Action Recognition (BAR) aims to recognize human actions directly from compressed image bitstreams without any pixel decoding. By operating entirely in byte space, BAR is inherently independent of file integrity and pixel-level reconstruction, making it naturally applicable to privacy-sensitive scenarios and robust against bitstream corruption. In this paper, we propose ByteAction, a BAR foundation model that achieves accurate action recognition on corrupted image bitstreams. ByteAction follows a dual-view byte-level recognition framework. It constructs weakly and strongly corrupted bitstream views, which are augmented by Bitstream Pattern Augmentation (BPA) and encoded with a shared ByteFormer backbone. The model is optimized with both classification and corruption consistency objectives. Specifically, we propose Bitstream Pattern Augmentation (BPA), which reshapes one-dimensional byte sequences into two-dimensional byte matrix and applies region-level erasure to encourage the model to learn robust cross-region byte dependencies. We further propose a Corruption Consistency Training strategy that constrains the model to maintain stable predictions across different corruption severities through bidirectional KL divergence. Experiments on the image bitstream from Stanford40, PPMI, and PASCAL VOC 2012 Action demonstrate that ByteAction achieves state-of-the-art corruption robustness across all scenarios while maintaining competitive intact bitstream performance.