Algorithmic Bias and Fairness

Algorithmic bias occurs when a computational system produces or reinforces systematically different outcomes that are harmful, unjustified, or poorly aligned with its intended purpose. Bias can enter through the problem definition, data collection, labels, features, model objective, decision threshold, user interface, or the institutional process surrounding the model. It is not limited to a programmer expressing prejudice, and removing a protected attribute does not remove information correlated with that attribute. A system can be accurate on average while failing particular groups or intersections of groups.

Fairness is a normative goal, not one universal statistic. Demographic parity asks whether groups receive a positive outcome at similar rates. Equal opportunity compares true positive rates for people who meet a relevant condition, while equalized odds also compares false positive rates. Calibration asks whether the same score has a similar meaning across groups. These criteria answer different moral and practical questions, and they can conflict when groups have different observed base rates or when labels reflect unequal conditions. Selecting a metric therefore requires a justified account of the decision, the harms of different errors, and the people who bear them.

Evidence of disparity requires disaggregated evaluation. The Gender Shades study showed that commercial gender classification systems tested on a more balanced benchmark had much larger error rates for darker-skinned women than for lighter-skinned men. The lesson is broader than facial analysis: aggregate performance can hide intersectional failures, and benchmark composition shapes what developers notice. Testing should examine false positives, false negatives, calibration, uncertainty, missing data, accessibility, and performance across relevant subgroups. Small sample sizes and unstable categories also need to be reported rather than converted into confident claims.

Technical mitigation can include improving data quality and coverage, revising labels or objectives, reweighting examples, constraining a model, adjusting thresholds, or changing how outputs are used. None of these steps proves that a system is fair. Fairness and Abstraction in Sociotechnical Systems warns that technical work can fail when it ignores the social process around a model, treats people as fixed categories, or assumes the modeled task is the real problem. Governance therefore includes affected-community input, documentation, contestability, monitoring after deployment, and the option not to automate.

In education, learners can audit a small fictional admissions or support model. They should define the decision, calculate a confusion matrix for each subgroup, compare two fairness criteria, and identify who benefits or is harmed by each choice. They can then propose a non-model change, such as improving access to support or adding human review. Schools should not use a fairness score as permission for high-stakes automation. They need lawful purposes, data minimization, accessible explanations, appeal routes, and evidence that the overall process supports learners. Responsible practice asks not only whether predictions are statistically balanced, but whether the task, data, intervention, and accountability structure are educationally defensible.