Data Leakage in Machine Learning

Data leakage occurs when model development uses information that would not legitimately be available for the intended prediction. It can make an evaluation look impressive while giving a misleading picture of real use. Leakage does not require copying the answer directly into a training column. It can enter through timing, duplicated records, preprocessing, or relationships among observations that a simple random split fails to respect.

Begin with the prediction moment. Suppose a school wants a model to identify, in week three, learners who may need additional course support. A dataset includes a field recording whether a learner completed the final examination. That field contains information from after week three and cannot be used for the stated task. A model that relies on it may perform well in a retrospective spreadsheet but cannot offer the promised early support. Removing names from the data would not fix this timing problem.

A second route involves related records. If several drafts of one learner's essay appear in both training and test sets, the model may benefit from having seen almost the same writing. Randomly splitting rows does not create independent learners. If the intended use is prediction for new learners, keep each learner's records together. If the claim concerns new schools or future terms, the split may need to respect schools or chronology as well. The correct unit follows the intended generalization claim.

Preprocessing can leak information too. Estimating a mean for missing-value imputation, selecting predictive features, or fitting a dimensionality-reduction transformation using the entire dataset allows test information to influence the model. Fit these operations only on the training portion, then apply the learned transformation to held-out cases. During cross-validation, repeat that separation within each fold. A pipeline can enforce the sequence, but only if the data boundaries and operations have been specified correctly.

Research by Kapoor and Narayanan documents leakage as a serious source of reproducibility problems in machine-learning-based science. For educators, the lesson is to ask how an impressive score was produced. Request a data timeline, a description of related observations, the split rules, and the order of preprocessing and model selection. Repeatedly trying settings against the final test set also turns that set into development information. A nominally untouched file name is not evidence of an independent evaluation.

A useful classroom exercise provides a fictional feature list for the week-three support model. Learners mark which fields were available at prediction time, group records belonging to the same person, and redesign the evaluation. They then explain why the corrected score might be lower yet more informative. Leakage prevention does not guarantee a useful or fair model, but it makes the evidence more credible. The practical standard is that evaluation should reproduce the information and independence constraints of the decision the institution actually intends to make.