11.6

Consider a classification problem where the classifier predicts whether a person has a particular disease. Suppose that 95% of the people tested do not suffer from the disease. Let pos denote the fraction of true positives, which is 5% of the test cases, and let neg denote the fraction of true negatives, which is 95% of the test cases. Consider the following classifiers:

  • Classifier \(C_1\), which always predicts negative (a rather useless classifier, of course)

  • Classifier \(C_2\), which predicts positive in 80% of the cases where the person actually has the disease but also predicts in 5% of the cases where the person does not have the disease.

  • Classifier \(C_3\), which predicts positive in 95% of the cases where the person actually has the disease but also predicts positive in 20% of the cases where the person does not have the disease.

For each classifier, let t_pos denote the true positive fraction, that is the fraction of cases where the classifier prediction was positive, and the person actually had the disease. Let f_pos denote the false positive fraction, that is the fraction of cases where the prediction was positive, but the person did not have the disease. Let t_neg denote true negative and f_neg denote false negative fractions, which are defined similary, but for the cases where the classifier prediction was negative.

  1. Compute the following metrics for each classifier:

  2. Accuracy, defined as (t_pos + t_neg) / (pos + neg), that is, the fraction of the time when the classifier gives the correct classification.

  1. Recall (also known as sensitivity) defined as t_pos / pos, that is, how many of the actual positive cases are classified as positive.

  2. Precision, defined as t_pos/(t_pos + f_pos), that is, how often the positive prediction is correct.

  3. Specificity, defined as t_neg/neg.

  1. If you intend to use the results of classification to perform further screening for the disease, how would you choose between the classifiers?

  2. On the other hand, if you intend to use the result of classification to start medication, where the medication could have harmful effects if given to someone who does not have the disease, how would you choose between the classifiers?

// TODO.