Nonstandard English and the Automated Scoring of Open-Ended Math Problems

An open-ended mathematics answer should be judged for its reasoning, without an irrelevant penalty for the variety of English a learner uses. An Educational Data Mining paper examines this problem with authentic student responses and GPT-4o. Its results reveal sensitivity to the grading prompt, but they do not provide a simple verdict that the model is either biased or fair across language varieties.

Abubakir Siedahmed and colleagues selected responses from ASSISTments containing four grammatical features associated with African American Vernacular English: negative concord, subject-verb agreement leveling, preterit done, and habitual be. After cleaning and expert checking, the analysis included 394 responses across 349 mathematics problems from seventy-five schools. These features can also occur in other varieties of English. Individual student demographics were unavailable, so the study cannot identify the race or dialect identity of each response's author.

For each original response, the researchers constructed a comparison version by standardizing the targeted grammatical feature. This is a paired text comparison, not a comparison between separate groups of students. Other dialect features, spelling differences, or incomplete phrasing could remain in both versions. Preserving most of the original response helps keep the mathematical content comparable, but it also means that the comparison is not between fully standardized and wholly dialectal language.

The model scored both versions using two prompts and several temperature settings. One prompt gave grading instructions without referring to dialect. The other explicitly asked the model to treat standard English and dialect varieties equally. The principal paired comparisons found no significant difference between the original and modified responses under either prompt. That result does not establish equivalence or comprehensive fairness; the manipulation was limited, and some linguistic categories had relatively few examples.

Changing the prompt raised scores for both text versions. The reported mean increase was 0.34 points for original responses and 0.26 points for the comparison responses on the zero-to-four scale. However, the published prompts contain another difference that matters to interpretation: the highest-score criterion in the first prompt required a complete explanation or justification, while the second omitted that phrase. Our reading of the displayed prompt table therefore identifies a possible scoring-rule confound. The observed increase cannot be cleanly attributed to the instruction about dialect alone.

For assessment teams, the useful lesson is methodological. A test of language-related scoring should hold the mathematical rubric constant, preserve the intended meaning of paired answers, and compare model decisions with careful human judgments. Raising scores is not automatically improving fairness or accuracy. An evaluation also needs to identify whether correct reasoning is recognized and whether the system applies the same substantive expectations across ways of expressing it.

The study opens a valuable line of inquiry in content assessment, beyond the more familiar case of essay grading. Its scope remains one model, selected linguistic features, and a restricted sample of mathematics responses. It did not measure learning gains or validate an operational grading policy. The paper's contribution is to make language variation and prompt sensitivity visible in a domain where educators might otherwise assume that only the mathematics matters.