FuTH-Net: Fusing Temporal Relations and Holistic Features for Aerial Video Classification

Pu Jin, Lichao Mou, Yuansheng Hua, Gui Song Xia, Xiao Xiang Zhu

Research output: Contribution to journalArticlepeer-review

7 Scopus citations

Abstract

Unmanned aerial vehicles (UAVs) are now widely applied to data acquisition due to its low cost and fast mobility. With the increasing volume of aerial videos, the demand for automatically parsing these videos is surging. To achieve this, current research mainly focuses on extracting a holistic feature with convolutions along both spatial and temporal dimensions. However, these methods are limited by small temporal receptive fields and cannot adequately capture long-term temporal dependencies that are important for describing complicated dynamics. In this article, we propose a novel deep neural network, termed Fusing Temporal relations and Holistic features for aerial video classification (FuTH-Net), to model not only holistic features but also temporal relations for aerial video classification. Furthermore, the holistic features are refined by the multiscale temporal relations in a novel fusion module for yielding more discriminative video representations. More specially, FuTH-Net employs a two-pathway architecture: 1) a holistic representation pathway to learn a general feature of both frame appearances and short-term temporal variations and 2) a temporal relation pathway to capture multiscale temporal relations across arbitrary frames, providing long-term temporal dependencies. Afterward, a novel fusion module is proposed to spatiotemporally integrate the two features learned from the two pathways. Our model is evaluated on two aerial video classification datasets, ERA and Drone-Action, and achieves the state-of-the-art results. This demonstrates its effectiveness and good generalization capacity across different recognition tasks (event classification and human action recognition). To facilitate further research, we release the code at https://gitlab.lrz.de/ai4eo/reasoning/futh-net.

Original languageEnglish
Article number5618913
JournalIEEE Transactions on Geoscience and Remote Sensing
Volume60
DOIs
StatePublished - 2022
Externally publishedYes

Keywords

  • Aerial video classification
  • convolutional neural networks (CNNs)
  • holistic features
  • temporal relations
  • two-pathway
  • unmanned aerial vehicle (UAV)

Fingerprint

Dive into the research topics of 'FuTH-Net: Fusing Temporal Relations and Holistic Features for Aerial Video Classification'. Together they form a unique fingerprint.

Cite this