Spatio-Temporal Multimedia Big Data Analytics Using Deep Neural Networks

Authors

Samira Pouyanfar, Florida International UniversityFollow

Document Type

Dissertation

Degree

Doctor of Philosophy (PhD)

Major/Program

Computer Science

First Advisor's Name

Shu-Ching Chen

First Advisor's Committee Title

Committee Chair

Second Advisor's Name

Sitharama S. Iyengar

Second Advisor's Committee Title

Committee Member

Third Advisor's Name

Jainendra K Navlakha

Third Advisor's Committee Title

Committee Member

Fourth Advisor's Name

Xudong He

Fourth Advisor's Committee Title

Committee Member

Fifth Advisor's Name

Keqi Zhang

Fifth Advisor's Committee Title

Committee Member

Keywords

Multimedia information management, deep neural networks, big data, spatio-temporal data, multimodal deep learning

Date of Defense

6-6-2019

Abstract

With the proliferation of online services and mobile technologies, the world has stepped into a multimedia big data era, where new opportunities and challenges appear with the high diversity multimedia data together with the huge amount of social data. Nowadays, multimedia data consisting of audio, text, image, and video has grown tremendously. With such an increase in the amount of multimedia data, the main question raised is how one can analyze this high volume and variety of data in an efficient and effective way. A vast amount of research work has been done in the multimedia area, targeting different aspects of big data analytics, such as the capture, storage, indexing, mining, and retrieval of multimedia big data. However, there is insufficient research that provides a comprehensive framework for multimedia big data analytics and management.

To address the major challenges in this area, a new framework is proposed based on deep neural networks for multimedia semantic concept detection with a focus on spatio-temporal information analysis and rare event detection. The proposed framework is able to discover the pattern and knowledge of multimedia data using both static deep data representation and temporal semantics. Specifically, it is designed to handle data with skewed distributions. The proposed framework includes the following components: (1) a synthetic data generation component based on simulation and adversarial networks for data augmentation and deep learning training, (2) an automatic sampling model to overcome the imbalanced data issue in multimedia data, (3) a deep representation learning model leveraging novel deep learning techniques to generate the most discriminative static features from multimedia data, (4) an automatic hyper-parameter learning component for faster training and convergence of the learning models, (5) a spatio-temporal deep learning model to analyze dynamic features from multimedia data, and finally (6) a multimodal deep learning fusion model to integrate different data modalities. The whole framework has been evaluated using various large-scale multimedia datasets that include the newly collected disaster-events video dataset and other public datasets.

Identifier

FIDC007767

Recommended Citation

Pouyanfar, Samira, "Spatio-Temporal Multimedia Big Data Analytics Using Deep Neural Networks" (2019). FIU Electronic Theses and Dissertations. 4265.
https://digitalcommons.fiu.edu/etd/4265