Abstract:
Problem: Monitoring microservices in an environment with many interactions between
services and a tremendous amount of log data can be a challenge. The traditional methods have
the difficulty of finding new or subtle anomaly, and those aren’t real time. Addressing these
challenges is the goal of this project, which builds upon the development of a machine learning
based anomaly detection system intended to enhance both accuracy and adaptability when
working with microservices that are dynamically constructed.
Methodology: Machine learning methods are used to analyse log data from microservices in
this project, the combination of supervised and unsupervised learning approaches. It prepares
data through log parsing, tokenization and sequence extraction before analysing. Deep learning
tools, autoencoders, are used by the model to discover patterns and infer anomalies and based
on those reports are generated.
Results: The model for anomaly detection with LSTM yields accuracy of 94% on the test
dataset, having the weighted precision of 94.73% and recall of 93.55% and F1 score of 93.95%.
The model is able to account for sequential dependencies in the log data and demonstrates a
good level of generalization while overfitting is kept at a minimum. Its performance is good
over the major classes like CRITICAL, SEVERE, WARNING and NORMAL class with high
precision and recall. Though, false positives and false negatives are in the acceptable limits.
