Machine Learning Methods
Matthews correlation coefficient gives a more truthful score than accuracy or F1 score for binary classification tests, a 2020 study concludes
Binary classification is the task of putting each case into one of two groups. In a 2020 paper in BMC Genomics, Davide Chicco and Giuseppe Jurman report that two common scores for this task, accuracy and F1 score, can look overly positive on imbalanced datasets, where one group is much larger than the other. They conclude that the Matthews correlation coefficient, or MCC, gives a more informative and truthful score. A high MCC value appears only when the predictions do well across all four confusion-matrix categories, true positives, false negatives, true negatives, and false positives, and in proportion to the size of both groups.
Why do accuracy and F1 score mislead?
Accuracy and F1 score are among the most popular measures for binary classification, the authors write. They say these scores can look overly positive, especially on imbalanced datasets, where one group is much larger than the other.
How does MCC judge all four outcome categories?
The authors report that MCC produces a high score only if the prediction obtained good results in all four confusion-matrix categories: true positives, false negatives, true negatives, and false positives. The score also reflects the size of the positive and negative groups in the dataset. For that reason, the authors conclude that MCC is a more informative and truthful measure than accuracy or F1 score.
What did the authors examine?
The authors first explain the mathematical properties of MCC. Then they show its advantages in six synthetic use cases and in a real genomics scenario.
FAQ
What is binary classification?
Binary classification is a task that places each case into one of two groups. Its results are usually summarized in a confusion matrix with four outcomes: true positives, false negatives, true negatives, and false positives.
Why are accuracy and F1 score not enough?
The authors say these common scores can produce overly optimistic results, especially on imbalanced data. MCC avoids this by requiring good results in all four confusion-matrix categories before it gives a high score.
Should researchers stop using accuracy and F1 score?
The authors believe the Matthews correlation coefficient should be preferred over accuracy and F1 score when evaluating binary classification tasks.
Limits and what's next
The comparison uses six synthetic use cases and one real genomics scenario. The abstract reports no independent replication by another team. The authors present their conclusion as a recommendation. The authors state they believe MCC should be preferred over accuracy and F1 score by all scientific communities when evaluating binary classification tasks.
Source: Davide Chicco and Giuseppe Jurman, “The advantages of the Matthews correlation coefficient (MCC) over F1 score and accuracy in binary classification evaluation,” BMC Genomics, 2020.