Flood Management Platform for Multi-Dimensional Data Exploration and Big Data Analytics Based on Crowdsourcing

Abstract

"Flood disaster management critically depends on analyzing large amounts of data and integrating and processing data from multiple sources for effective decision-making and response. Without a specialized platform for flood data integration and visualization, managing the catastrophic impacts of floods efficiently is nearly impossible. This research addresses this challenge by developing a flood disaster data integration and analytics platform using big data analytics and advanced data visualization techniques. The flood disaster analytics platform was developed and implemented using the Agile development methodology. The framework is designed to use Apache Spark for streaming flood data processing, Hadoop HDFS for persistent storage, Jupyter Notebook for interactive exploration of the processed data, and Streamlit for generating intuitive web-based visualization interfaces. The platform processes large volumes of flood data from multiple sources, cleans and aggregates it using Apache Spark, and instantly visualizes the output through optimized Streamlit plots and visualization dashboards. This enables decision-makers, such as government leadership and disaster response authorities, to gain meaningful insights to act on big data in real-time. Testing of the flood data analytics platform showed high efficiency and reliability in handling large datasets, essential for flood disaster management. The platform-maintained data consistency with zero rows having negative values and achieved rapid execution times of 0.04 seconds for full datasets, enabling near real-time processing. Scalability tests demonstrated stable performance across various data sizes, with times between 0.01 and 0.02 seconds. Resource utilization was optimized, as observed in the Spark UI, and fault tolerance was confirmed through effective checkpointing and recovery. These results indicate the platform's robustness in managing flood data and providing timely insights for decision-making. Subject Descriptions: • Information systems → Data analytics → Big data processing • Human-centered computing → Human-computer interaction (HCI) → Data visualization interfaces"