# Dense Multitask Learning to Reconfigure Comics

arXiv: https://arxiv.org/abs/2307.08071 · HTML: https://mlsanity.com/paper/2307.08071/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 52 references that are in the map.
- First posted on arXiv 16 Jul 2023.

## Builds on (most cited first, 29 of 29)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Conditional Generative Adversarial Nets](https://mlsanity.com/paper/1411.1784/) (2014)
- [Decoupled Weight Decay Regularization](https://mlsanity.com/paper/1711.05101/) (2017)
- [Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformers](https://mlsanity.com/paper/2012.15840/) (2020)
- [Pre-Trained Image Processing Transformer](https://mlsanity.com/paper/2012.00364/) (2020)
- [A Survey on Multi-Task Learning](https://mlsanity.com/paper/1707.08114/) (2017)
- [GradNorm: Gradient Normalization for Adaptive Loss Balancing in Deep Multitask Networks](https://mlsanity.com/paper/1711.02257/) (2017)
- [Multimodal Unsupervised Image-to-Image Translation](https://mlsanity.com/paper/1804.04732/) (2018)
- [Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks](https://mlsanity.com/paper/1703.10593/) (2017)
- [Taskonomy: Disentangling Task Transfer Learning](https://mlsanity.com/paper/1804.08328/) (2018)
- [Transformer Interpretability Beyond Attention Visualization](https://mlsanity.com/paper/2012.09838/) (2020)
- [End-to-End Multi-Task Learning with Attention](https://mlsanity.com/paper/1803.10704/) (2018)
- [Which Tasks Should Be Learned Together in Multi-task Learning?](https://mlsanity.com/paper/1905.07553/) (2019)
- [Sketch-based Manga Retrieval using Manga109 Dataset](https://mlsanity.com/paper/1510.04389/) (2015)
- [Diverse Image-to-Image Translation via Disentangled Representations](https://mlsanity.com/paper/1808.00948/) (2018)
- [UniT: Multimodal Multitask Learning with a Unified Transformer](https://mlsanity.com/paper/2102.10772/) (2021)
- [Robust Learning Through Cross-Task Consistency](https://mlsanity.com/paper/2006.04096/) (2020)
- [TVT: Transferable Vision Transformer for Unsupervised Domain Adaptation](https://mlsanity.com/paper/2108.05988/) (2021)
- [Many Task Learning with Task Routing](https://mlsanity.com/paper/1903.12117/) (2019)
- [MulT: An End-to-End Multitask Learning Transformer](https://mlsanity.com/paper/2205.08303/) (2022)
- [Towards Instance-level Image-to-Image Translation](https://mlsanity.com/paper/1905.01744/) (2019)
- [Cycle-IR: Deep Cyclic Image Retargeting](https://mlsanity.com/paper/1905.03556/) (2019)
- [Unsupervised learning for cross-domain medical image synthesis using deformation invariant cycle consistency networks](https://mlsanity.com/paper/1808.03944/) (2018)
- [Spatio-Temporal Multi-Task Learning Transformer for Joint Moving Object Detection and Segmentation](https://mlsanity.com/paper/2106.11401/) (2021)
- [Estimating Image Depth in the Comics Domain](https://mlsanity.com/paper/2110.03575/) (2021)

## Built on by (most cited first, 1 of 1)

- [One missing piece in Vision and Language: A Survey on Comics Understanding](https://mlsanity.com/paper/2409.09502/) (2024)

## Abstract (arXiv)

In this paper, we develop a MultiTask Learning (MTL) model to achieve dense predictions for comics panels to, in turn, facilitate the transfer of comics from one publication channel to another by assisting authors in the task of reconfiguring their narratives. Our MTL method can successfully identify the semantic units as well as the embedded notion of 3D in comic panels. This is a significantly challenging problem because comics comprise disparate artistic styles, illustrations, layouts, and object scales that depend on the authors creative process. Typically, dense image-based prediction techniques require a large corpus of data. Finding an automated solution for dense prediction in the comics domain, therefore, becomes more difficult with the lack of ground-truth dense annotations for the comics images. To address these challenges, we develop the following solutions: 1) we leverage a commonly-used strategy known as unsupervised image-to-image translation, which allows us to utilize a large corpus of real-world annotations; 2) we utilize the results of the translations to develop our multitasking approach that is based on a vision transformer backbone and a domain transferable attention module; 3) we study the feasibility of integrating our MTL dense-prediction method with an existing retargeting method, thereby reconfiguring comics.