Precision, Recall, and Decision Thresholds

A classifier often produces a score before a system decides whether to flag a case. A decision threshold turns that score into an action. Lowering the threshold usually flags more cases; raising it usually flags fewer. The choice matters because a missed case and an unnecessary flag can have different consequences. Precision and recall help educators describe that tradeoff more clearly than a single overall accuracy figure.

Precision asks: among the cases the system flagged, how many truly belonged to the target category? Recall asks: among all cases that truly belonged to the target category, how many did the system flag? Both depend on a clearly defined positive category and a credible reference judgment. They are not interchangeable measures of correctness. A system can achieve high precision by flagging very few easy cases while missing many that need attention.

Suppose a fictional writing-feedback tool reviews 100 essays. Human reviewers identify 20 that need feedback on evidence use. The tool flags 25 essays, of which 15 actually need that feedback. Its precision is 15 divided by 25, or 60 percent. Its recall is 15 divided by 20, or 75 percent. There are ten unnecessary flags and five missed cases. These counts make the practical workload visible: the teacher inspects 25 essays, while some unflagged essays still need support.

Changing the threshold alters these counts without necessarily changing the model's underlying scores. A lower threshold may be appropriate when review is quick and missing a learning need is costly. A higher threshold may be necessary when every flag initiates a burdensome process. Such choices require discussion with the people affected. A threshold that is reasonable for suggesting optional practice may be unacceptable for an allegation of misconduct or a decision about access to a course. The same score does not justify the same action everywhere.

Precision–recall curves show performance across thresholds. They help teams inspect the range of possible policies rather than report only a favorable point. The F1 score combines precision and recall through a harmonic mean, but its equal weighting is not an ethical or educational rule. Where consequences differ, report the underlying counts and use a decision criterion that reflects the actual purpose. Precision also changes with the prevalence of the target condition, so results from a specially balanced test set may not describe normal classroom use.

Thresholds should be selected using development data and then tested on independent, representative cases. Inspect uncertainty, variation across learner groups, and disagreement in human labels. A well-chosen threshold cannot repair a poorly defined construct or an unreliable model. In teaching, let learners calculate the essay example and compare alternative thresholds under a fixed review capacity. Ask them to justify the resulting policy in ordinary language. The goal is to understand that an automated flag is a designed decision with costs, benefits, and limits that the institution must make explicit.