ATMOSYNC - Music Recommendation system based on Room Ambiance

Abstract

"This document introduces a solution to the challenge of static music recommendations in dynamic environments, particularly in crowded gatherings. Traditional music recommendation systems (MRS) often fall short in these settings by neglecting real-time context, such as the ambiance in a room, leading to suboptimal user experiences. To address this issue, a novel MRS is proposed, leveraging both video and audio inputs to comprehend the ambiance and deliver context-aware music suggestions. The methodology first uses image classification with 3D Convolutional Neural Networks (CNNs) exclusively on image frames to identify ambient illumination and room settings. This method combines image processing and audio analysis. To capture the temporal spectrum aspects of audio data, it is simultaneously transformed into Mel spectrograms and Mel-frequency cepstral coefficients (MFCCs). To combine visual and auditory information, these elements from the image frames and audio are then early fused. The atmosphere of the room is ascertained by further layers processing the fused information, which determines the suggested music selection based on this analysis. The initial prototype demonstrates promising results in ambiance detection, and as expected, due to dataset limitations. This project represents a significant advancement in MRS by incorporating visual and auditory elements. Future enhancements are planned to further refine the model's performance."