Novel Evaluation Framework for Machine Learning Considering External Factors

Abstract

"The task of model evaluation is a vital step in the development of machine learning models, but it can be noticed that despite its importance, all current standardized evaluation metrics for machine learning models are solely focused on the direct performance of a model and fail to provide insight on any external factors that will contribute to the model's performance. Therefore, in this paper, thorough research will be conducted on the creation and justification of a new evaluation framework that utilizes external factors in order to provide a useful contribution for researchers to gain clearer insight into the non-performance aspects of the machine learning models. To achieve this, the task was divided into key segments. Primarily, research into the identification of these external factors was conducted. Once the factors were identified evaluation metrics were formulated which incorporated both internal performance factors with the chosen external factors to allow interpretable quantification of the results. This was done for both classification and regression models to allow more varied evaluation. Test results show the metrics' efficacy as they clearly show the impact of various external factors on the resulting output whereas conventional performance metrics fail to make the distinction between the model's difference. This is clear for every metric created in both classification and regression-based models."