Precision and recall

In a classification task, the precision for a class is the number of true positives (i.e. the number of items correctly labelled as belonging to the positive class) divided by the total number of elements labelled as belonging to the positive class (i.e. the sum of true positives and false positives, which are items incorrectly labelled as belonging to the class). Recall in this context is defined as the number of true positives divided by the total number of elements that actually belong to the positive class (i.e. the sum of true positives and false negatives, which are items which were not labelled as belonging to the positive class but should have been). Precision and recall are not particularly useful metrics when used in isolation. For instance, it is possible to have perfect recall by simply retrieving every single item. Likewise, it is possible to achieve perfect precision by selecting only a very small number of extremely likely items. In a classification task, a precision score of 1.0 for a class C means that every item labelled as belonging to class C does indeed belong to class C (but says nothing about the number of items from class C that were not labelled correctly) whereas a recall of 1.0 means that every item from class C was labelled as belonging to class C (but says nothing about how many items from other classes were incorrectly also labelled as belonging to class C). Often, there is an inverse relationship between precision and recall, where it is possible to increase one at the cost of reducing the other, but context may dictate if one is more valued in a given situation: A smoke detector is generally designed to commit many Type I errors (to alert in many situations when there is no danger), because the cost of a Type II error (failing to sound an alarm during a major fire) is prohibitively high. As such, smoke detectors are designed with recall in mind (to catch all real danger), even while giving little weight to the losses in precision (and making many false alarms). In the other direction, Blackstone's ratio, "It is better that ten guilty persons escape than that one innocent suffer," emphasizes the costs of a Type I error (convicting an innocent person). As such, the criminal justice system is geared toward precision (not convicting innocents), even at the cost of losses in recall (letting more guilty people go free). A brain surgeon removing a cancerous tumor from a patient's brain illustrates the tradeoffs as well: The surgeon needs to remove all of the tumor cells since any remaining cancer cells will regenerate the tumor. Conversely, the surgeon must not remove healthy brain cells since that would leave the patient with impaired brain function. The surgeon may be more liberal in the area of the brain they remove to ensure they have extracted all the cancer cells. This decision increases recall but reduces precision. On the other hand, the surgeon may be more conservative in the brain cells they remove to ensure they extract only cancer cells. This decision increases precision but reduces recall. That is to say, greater recall increases the chances of removing healthy cells (negative outcome) and increases the chances of removing all cancer cells (positive outcome). Greater precision decreases the chances of removing healthy cells (positive outcome) but also decreases the chances of removing all cancer cells (negative outcome). Usually, precision and recall scores are not discussed in isolation. A precision-recall curve plots precision as a function of recall; usually precision will decrease as the recall increases. Alternatively, values for one measure can be compared for a fixed level at the other measure (e.g. precision at a recall level of 0.75) or both are combined into a single measure. Examples of measures that are a combination of precision and recall are the F-measure (the weighted harmonic mean of precision and recall), or the Matthews correlation coefficient, which is a geometric mean of the chance-corrected variants: the regression coefficients Informedness (DeltaP') and Markedness (DeltaP). Accuracy is a weighted arithmetic mean of Precision and Inverse Precision (weighted by Bias) as well as a weighted arithmetic mean of Recall and Inverse Recall (weighted by Prevalence). and their geometric mean Matthews correlation coefficient thus acts like a debiased F-measure. == Definition ==

For classification tasks, the terms true positives, true negatives, false positives, and false negatives compare the results of the classifier under test with trusted external judgments. The terms positive and negative refer to the classifier's prediction (sometimes known as the expectation), and the terms true and false refer to whether that prediction corresponds to the external judgment (sometimes known as the observation). Let us define an experiment from P positive instances and N negative instances for some condition. The four outcomes can be formulated in a 2×2 contingency table or confusion matrix, as follows: Precision and recall are then defined as: \begin{align} \text{Precision} &= \frac{tp}{tp + fp} \\ \text{Recall} &= \frac{tp}{tp + fn} \, \end{align} Recall in this context is also referred to as the true positive rate or sensitivity, and precision is also referred to as positive predictive value (PPV); other related measures used in classification include true negative rate and accuracy. Probabilistic definition Precision and recall can be interpreted as (estimated) conditional probabilities: Precision is given by \mathbb{P}(C=P|\hat{C}=P) while recall is given by \mathbb{P}(\hat{C}=P|C=P), where \hat{C} is the predicted class and C is the actual class (i.e. C=P means the actual class is positive). Both quantities are, therefore, connected by Bayes' theorem. No-skill classifiers The probabilistic interpretation allows to easily derive how a no-skill classifier would perform. A no-skill classifier is defined by the property that the joint probability \mathbb{P}(C=P,\hat{C}=P)=\mathbb{P}(C=P)\mathbb{P}(\hat{C}=P) is just the product of the unconditional probabilities since the classification and the presence of the class are independent. For example the precision of a no-skill classifier is simply a constant \mathbb{P}(C=P|\hat{C}=P)=\frac{\mathbb{P}(C=P,\hat{C}=P)}{\mathbb{P}(\hat{C}=P)}=\mathbb{P}(C=P), i.e. determined by the probability/frequency with which the class P occurs. A similar argument can be made for the recall: \mathbb{P}(\hat{C}=P|C=P)=\frac{\mathbb{P}(C=P,\hat{C}=P)}{\mathbb{P}(C=P)}=\mathbb{P}(\hat{C}=P) which is the probability for a positive classification. ==Imbalanced data==

\text{Accuracy}=\frac{TP+TN}{TP+TN+FP+FN} \, Accuracy can be a misleading metric for imbalanced data sets. Consider a sample with 95 negative and 5 positive values. Classifying all values as negative in this case gives 0.95 accuracy score. There are many metrics that don't suffer from this problem. For example, balanced accuracy (bACC) normalizes true positive and true negative predictions by the number of positive and negative samples, respectively, and divides their sum by two: \text{Balanced accuracy}= \frac{TPR + TNR}{2}\, For the previous example (95 negative and 5 positive samples), classifying all as negative gives 0.5 balanced accuracy score (the maximum bACC score is one), which is equivalent to the expected value of a random guess in a balanced data set. Balanced accuracy can serve as an overall performance metric for a model, whether or not the true labels are imbalanced in the data, assuming the cost of FN is the same as FP. The TPR and FPR are a property of a given classifier operating at a specific threshold. However, the overall number of TPs, FPs etc depend on the class imbalance in the data via the class ratio r = P/N. As the recall (or TPR) depends only on positive cases, it is not affected by r, but the precision is. We have that \text{Precision} = \frac{TP}{TP+FP} = \frac{P \cdot TPR}{P \cdot TPR+ N \cdot FPR} = \frac{TPR}{TPR+ \frac{1}{r} FPR}. Thus the precision has an explicit dependence on r. Starting with balanced classes at r =1 and gradually decreasing r, the corresponding precision will decrease, because the denominator increases. Another metric is the predicted positive condition rate (PPCR), which identifies the percentage of the total population that is flagged. For example, for a search engine that returns 30 results (retrieved documents) out of 1,000,000 documents, the PPCR is 0.003%. \text{Predicted positive condition rate}=\frac{TP+FP}{TP+FP+TN+FN} \, According to Saito and Rehmsmeier, precision-recall plots are more informative than ROC plots when evaluating binary classifiers on imbalanced data. In such scenarios, ROC plots may be visually deceptive with respect to conclusions about the reliability of classification performance. Different from the above approaches, if an imbalance scaling is applied directly by weighting the confusion matrix elements, the standard metrics definitions still apply even in the case of imbalanced datasets. The weighting procedure relates the confusion matrix elements to the support set of each considered class. == F-measure ==

A measure that combines precision and recall is the harmonic mean of precision and recall, the traditional F-measure or balanced F-score: F = 2 \cdot \frac{\mathrm{precision} \cdot \mathrm{recall}}{ \mathrm{precision} + \mathrm{recall}} This measure is approximately the average of the two when they are close, and is more generally the harmonic mean, which, for the case of two numbers, coincides with the square of the geometric mean divided by the arithmetic mean. There are several reasons that the F-score can be criticized, in particular circumstances, due to its bias as an evaluation metric. This is also known as the F_1 measure, because recall and precision are evenly weighted. It is a special case of the general F_\beta measure (for non-negative real values of \beta): F_\beta = (1 + \beta^2) \cdot \frac{\mathrm{precision} \cdot \mathrm{recall} }{ \beta^2 \cdot \mathrm{precision} + \mathrm{recall}} Two other commonly used F measures are the F_2 measure, which weights recall higher than precision, and the F_{0.5} measure, which puts more emphasis on precision than recall. The F-measure was derived by van Rijsbergen (1979) so that F_\beta "measures the effectiveness of retrieval with respect to a user who attaches \beta times as much importance to recall as precision". It is based on van Rijsbergen's effectiveness measure E_{\alpha} = 1 - \frac{1}{\frac{\alpha}{P} + \frac{1-\alpha}{R}}, the second term being the weighted harmonic mean of precision and recall with weights (\alpha, 1-\alpha). Their relationship is F_\beta = 1 - E_{\alpha} where \alpha=\frac{1}{1 + \beta^2}. ==Limitations as goals==

There are other parameters and strategies for performance metric of information retrieval system, such as the area under the ROC curve (AUC) or pseudo-R-squared. ==Multi-class evaluation==

Precision and recall values can also be calculated for classification problems with more than two classes. To obtain the precision for a given class, we divide the number of true positives by the classifier bias towards this class (number of times that the classifier has predicted the class). To calculate the recall for a given class, we divide the number of true positives by the prevalence of this class (number of times that the class occurs in the data sample). The class-wise precision and recall values can then be combined into an overall multi-class evaluation score, e.g., using the macro F1 metric. ==See also==