Assessment of Multimodal Rainfall Classification Systems Based on an Audio/Video Dataset

Abstract

In the past few years, there has been an increase in natural disasters due to hydrogeological instability caused by heavy rain. Therefore, to reduce the risk of an imminent occurrence of a disastrous event and reduce the risk to humans, an accurate estimate of the precipitation levels based on advanced machine learning techniques is necessary. In this paper, a new dataset is proposed containing audio/video data recorded via a multimodal rain gauge created ad hoc. The dataset, denominated AVDB-4RC (Audio/Video Database for Rainfall Classification), contains digital audio/video sequences recorded for seven different levels of precipitation intensity. In particular, the database presents a set of audio sequences containing the acoustic timbre produced by the rain and video sequences containing rain videos, both in seven different intensities, i.e., “No rain,†“Weak rain,†“Moderate rain,†“Heavy rain†and “Very heavy rain,†"Shower rain" and "Cloudburst rain." For the validation of the dataset, the paper proposes a novel rainfall classification approach based on a video pattern recognition system that uses CNN neural networks. The average classification accuracy is approximately 49% and can reach 75% if the adjacent misclassifications are not considered. Presumably, it is the first open dataset from the new generation acoustic/video rain gauges available for evaluating the estimated rainfall performance. We hope that this new open dataset will encourage a comparison of rainfall estimation/classification algorithms on this common database so that the adopted techniques are objectively assessed and improved.