Five AI systems outscored the average OOP cohort but still failed compilation and advanced concepts

Marina Lepp and Joosep Kaimre revisit how current generative AI systems perform on authentic introductory object-oriented programming assessments. They evaluate ChatGPT-5.2, DeepSeek-V3, Gemini 2.5 Flash, Claude Sonnet 4.5, and Microsoft 365 Copilot using programming tests and examination tasks from a university course. Generated solutions are graded with the same criteria used for students and compared with historical student results and findings from the previous year.

All five systems scored above the average student cohort and often earned full marks on longer programming tasks. This is consequential for assessment design. A take-home task that once distinguished whether a novice could integrate several OOP ideas may now be completed convincingly by a widely available model. The paper also shows that high aggregate performance does not mean uniform capability. Some outputs failed to compile. Interfaces, abstract classes, inheritance-related problems, and graphics questions requiring image interpretation continued to produce errors.

Using authentic assessments and ordinary grading criteria is a strength because the evaluation speaks directly to existing course practice. The comparison across five systems also avoids treating one product as representative of all generative AI. Yet historical student averages and model scores answer different questions. Students worked under course conditions and learned over time; models responded to supplied tasks. A model's grade is not evidence that a learner who submits its output understands the code, can debug it, or can transfer the concepts to a new problem.

The study also requires temporal caution. Product names and versions will change, and prompts, tools, image inputs, sampling, or repeated attempts can alter results. Authentic tasks may enter future training data. A score from 2026 is therefore a snapshot rather than a permanent ranking. The paper identifies recurring failure patterns, but broader replication should record exact interfaces and settings, run several trials, test unseen tasks, and analyze why code fails instead of reporting only total marks.

For teaching, the findings support assessment redesign rather than a simple prohibition. Instructors can combine supervised coding, oral explanation, code tracing, debugging of unfamiliar defects, version-history review, and iterative projects tied to local decisions. Students may use AI in declared phases, then explain every design choice and complete a related task without assistance. Rubrics can reward tests, reasoning, maintainability, provenance, and response to feedback rather than only a final program.

For AIEDHK, the paper offers a current capability audit and a warning about construct validity. If an assessment intends to measure a student's independent OOP knowledge, an AI-generated solution can invalidate the inference even when the code is correct. If the goal includes responsible AI-supported development, the task should directly assess verification, debugging, attribution, and judgment. The observed strengths and compilation failures together make one principle clear: educators must assess the learner's accountable performance, not merely the polished artifact available at submission time. Updated tasks should be piloted for difficulty and accessibility so redesigned controls do not accidentally measure anxiety, typing speed, or prior tool access instead of programming knowledge.