JAutoPipe: Auto Scaling Framework for Pipeline Architecture

Abstract

With the advancement of technology, the data production rate has increased. In numerous domains of application, it is a critical necessity to process such data, in real-time rather than a store and process approach. When it comes to real-time processing, many of the applications adapt the pipeline architecture to process data in a streaming fashion. Despite its frequent utilization, the traditional pipeline suffers from a fundamental issue of performance bottlenecks due to its connected stages. This research, presents a solution in the form of a framework that aims at overcoming the abovementioned bottleneck of the pipeline architecture. The solution framework, analyses the stages in the pipeline, identifies the bottleneck and scales it appropriately in two different scaling methods, to overcome the issue. The framework had an overall accuracy of 87.5% and improved performance significantly for the identified use cases of the application.