Categorization of Disaster Related Tweets using Multimodal Approach

Abstract

Contents shared in form of text and images in multimedia during and after disasters can be used to analyze the information about the event. Report of affected people as missing or injured, infrastructure and utility damages, rescue and volunteering needed, not humanitarian or other relevant information can also be found with this analysis. It has been found that only few researches focuses on text as well as image modality for such analysis. Also no works has been done for mixture of dissimilar and similar category text-image pairs. In this paper, we aim to use both text as well as image of different category and fuse them using score fusion for joint representation of text and images. For text modality, we have used BERT model and for image modality we have used VGG16 modality and fused them using late fusion for multimodal analysis of disaster related tweet categorization.