Anomaly detection in microservices using machine learning techniques

Abstract

Problem: Monitoring microservices in an environment with many interactions between services and a tremendous amount of log data can be a challenge. The traditional methods have the difficulty of finding new or subtle anomaly, and those aren’t real time. Addressing these challenges is the goal of this project, which builds upon the development of a machine learning based anomaly detection system intended to enhance both accuracy and adaptability when working with microservices that are dynamically constructed. Methodology: Machine learning methods are used to analyse log data from microservices in this project, the combination of supervised and unsupervised learning approaches. It prepares data through log parsing, tokenization and sequence extraction before analysing. Deep learning tools, autoencoders, are used by the model to discover patterns and infer anomalies and based on those reports are generated. Results: The model for anomaly detection with LSTM yields accuracy of 94% on the test dataset, having the weighted precision of 94.73% and recall of 93.55% and F1 score of 93.95%. The model is able to account for sequential dependencies in the log data and demonstrates a good level of generalization while overfitting is kept at a minimum. Its performance is good over the major classes like CRITICAL, SEVERE, WARNING and NORMAL class with high precision and recall. Though, false positives and false negatives are in the acceptable limits.