This study presented and analyzed the effect on the rank of machine learning classification algorithms by switching the evaluation metrics. For the analysis, 12 classification machine learning algorithms were trained on 8 classification tasks taken from OpenML and their performances were evaluated using 5 evaluation metrics. Then the variance in ranks of each classifier was computed for each of the 8 tasks. The results showed that the variance of rank for Random forest and KNN was 0 for each of the 8 tasks indicating that their rank was unaffected even after changing the selected metric. In contrast algorithms such as AdaBoost, Logistic Regression, MLP, and support vector classifier displayed a higher variance in rank indicating that they were affected by changing the evaluation metric.