In a framework that an algorithm is supposed to predict "positive" or "negative". Some concepts are really confusing. So a summary here. All the concepts or metrics are widely used to measure the performance of the algorithm or machine learning model (which is essentially an computational intensive algorithm).
| ground truth\prediction |
positive |
negative |
rate |
| positive |
A |
B |
TPR |
| negative |
C |
D |
TNR |
| rate |
PPV,Precision |
NPV |
|
A: true positive (TP)
B: false negative (FN)
C: false positive (FP)
D: true negative (TN)
A+B: positive (P)
C+D: negative(N)
False reject rate (FRR) = B/(A+B) = FN/(TP+FN) = FN/P = 1-TPR
False accept rate (FAR) = C/(C+D) = FP/(FP+TN) = FPR
True positive rate (TPR) = A/(A+B) = TP/(TP+FN) = TP/P
False positive rate (FPR) = fall out = C/(C+D) = FP/(FP+TN) = FAR = 1-SPC
Accuracy (ACC) = (A+D)/(A+B+C+D)
Sensitivity = TPR = A/(A+B) = TP/(TP+FN)
Specificity (SPC) = TNR = D/(C+D) = TN/(FP+TN) = 1-FPR
Positive predictive value (PPV) = precision = A/(A+C) = TP/(TP+FP) = TP/P = TPR
Negative predictive value (NPV) = TN/(TN+FN)
False positive rate (FPR) = fall out = C/(C+D) = FP/(FP+TN) = FAR = 1-SPC
False discover rate (FDR) = C/(A+C) = FP/(TP+FP) = FP/P = 1-TPR = 1-PPV
*In bio-medical field, positive means disease, while negative indicates healthy.
Further more,
F1 Score, harmonic mean of precision and sensitivity
F1 = 2TP/(2TP+FP+FN)
Matthews correlation coefficient (MCC)
MCC = (TP*TN+FP*FN) / ((TP+FP)*(TP+FN)*(TN+FP)*(TN+FN))^0.5
When a model (classification model) is finalized, one may want to find an operating point, i.e., confidence threshold. This will be a trade-off between precision (higher with higher threshold) and recall (lower with higher threshold). Then you need a curve to show the performance at all possible threshold levels.
1. Receiver operating characteristic (ROC) curve is a plot of true positive ratio V.S. false positive ratio.
When compare the performance of two models, it is hard to tell which one is better given two curves. So people use the area under curve (AUC) to measure and compare performance (the larger the better). But as you can see, two different curves can have the same AUC. Then to choose which one is depends on whether high precision or high recall is more desirable.
The AUC can be any value between 0 and 1. A random guess classifier will create a straight line segment from (0, 0) and (1, 1). While it can happen that auc<0.5, but then one can flip the output prediction, to make a new classifier with auc' = (1-auc). In this sense, AUC can be considered as something equal or above 0.5.
In statistic learning, the AUC represents the probability that a model outputs higher score for a randomly chosen positive class than a randomly chosen negative class. To prove this, please refer to a very nice blog:
https://madrury.github.io/jekyll/update/statistics/2017/06/21/auc-proof.html.
2. Precision-Recall curve is, as the name says, a plot of precision ratio V.S. recall ratio.
references:
http://en.wikipedia.org/wiki/Sensitivity_and_specificity
