A primary mathematics AI pilot reported before-and-after gains, with comparative effectiveness still unclear

Primary-school word problems require students to connect language with a mathematical representation, choose a strategy and check whether an answer makes sense. Jingxi Liu and colleagues designed ChatGPT-MPS to support that process through structured guidance and opportunities to compare solutions. Their paper describes both system development and a classroom pilot. Reading the detailed analysis, however, is essential: the evidence it reports is narrower than the comparative effectiveness claim made in the abstract, and that distinction should shape any practical interpretation.

The study involved 104 fifth-grade pupils from two existing classes in Guangdong, China. One class of 52 used the system and another class of 52 served as a comparison. Assignment followed class membership rather than individual randomization. The AI group received a ten-minute introduction and worked for 40 minutes on twelve curriculum-aligned word problems with step-by-step support. The comparison class worked on the same problem set under the same time constraint without the system. The design therefore began with a potentially informative classroom comparison.

The main performance results nevertheless focus on the AI class alone. After removing three post-task outliers, the paired analysis included 49 pupils. Their mean score rose from 56.45 to 72.59, a gain of 16.14 points. This is a reported before-and-after change within the participating class. The main results do not present a clear between-group estimate supporting the abstract's statement that the experimental group improved more than the comparison group. A significant paired test cannot substitute for that comparison, because practice, instruction and other changes could also contribute to improvement.

The system itself offers useful design ideas. It organizes guidance for primary-level problems, preserves learning records and allows learners to access more than one solution method. Students also completed an interest questionnaire, whose results linked perceived value, knowledge and engagement with interest in the system. These are perceptions measured together, rather than independent evidence that the tool caused motivation or conceptual growth. Comparing approaches could support reasoning, but only if learners understand and can explain why a method fits the problem rather than merely read a generated procedure.

The study's short activity, single school and intact-class allocation restrict generalization. The removal of three observations also makes the reported analytic sample important. The paper does not establish delayed retention or performance on an unfamiliar problem without assistance, and the detailed results do not resolve comparative effectiveness. System responsiveness and positive interest ratings are relevant to feasibility, but cannot validate every generated explanation or demonstrate durable mathematical understanding. The appropriate conclusion is preliminary evidence of an immediate task-score change within a designed learning environment.

For AIEDHK, a follow-up evaluation could retain the system's structured guidance while reporting both classes' baseline and follow-up results transparently. A prespecified analysis should account for class allocation and document missing data or outlier decisions. Learners could also explain a solution, identify an incorrect step and solve a new problem without AI after a delay. These are proposed improvements beyond the reported pilot. The practical contribution is a reviewable design for supporting word-problem work, with a clear need for stronger evidence before claiming that the system outperforms conventional teaching or builds independent problem-solving skill.