Motivational agents changed some algebra-learning choices without a significant overall test advantage

Adding a friendly agent to an adaptive lesson does not automatically improve what students learn. Man Su and colleagues examined that assumption in an algebra environment that offered choices about optional learning activities. Their study treated those choices as part of self-regulation: when to attempt a task, seek support or move on. The main value of the investigation lies in combining outcome measures with interaction traces, while retaining an important null finding about the overall learning advantage of the agent condition.

Fifty-four ninth- and tenth-grade pupils from two German schools took part and were randomly assigned to versions of the AlgeSPACE system with or without motivational pedagogical agents. Five pupils were excluded because of incomplete post-tests or disengagement, leaving 49 in the analysis: 24 with agents and 25 without them. After a pre-test and brief tutorial, pupils worked independently for 25 minutes and completed an immediate post-test. The system recorded their choices, answers, solution steps and time spent during the activity.

The agents delivered motivational prompts around learning opportunities, such as trying a problem before receiving automated feedback. The study therefore examined a particular combination of visual presence and motivational messaging within a designed algebra system. It did not evaluate an open-ended generative tutor or compare every possible kind of adaptive feedback. The prompts were delivered at predefined points and were not dynamically personalized to each moment of misunderstanding or disengagement, a distinction relevant to interpreting both the outcomes and the proposed improvements.

Both conditions improved from pre-test to post-test. Although descriptive gains were larger with agents, the condition effect on post-test scores was not significant after adjustment for prior knowledge; the reported p value was .189. Error rates also did not differ significantly. Prior knowledge was a stronger predictor of performance. Process mining and sequence analyses revealed differences in how learners navigated optional activities, but such patterns should not be converted into a claim of proven overall effectiveness. A different path through a lesson can be informative without producing a reliably better test result.

The exploratory design places limits on subgroup and behavioral interpretations. The final sample was small, the activity brief, and all participants came from one national context and a narrow age range. Excluding some disengaged participants also matters when the intervention itself is intended to influence engagement. Logged decisions show what students selected, but do not fully reveal why they skipped a task or whether a choice was strategically appropriate. There was no delayed assessment of algebra understanding or evidence that these choice patterns would persist during repeated use.

For AIEDHK, the study suggests evaluating an agent through the learning decision it is intended to support. A pilot could ask learners to explain selected choices, such as skipping a familiar example or requesting another attempt, and compare those explanations with performance on a new problem. Repeated sessions and delayed tests would provide a stronger basis for judging benefit. These are proposed extensions beyond the experiment. The practical lesson is to distinguish an engaging interface, a meaningful learning choice and measurable understanding, while reporting null comparisons as clearly as positive interaction patterns.