A tailored mathematics chatbot increased situational interest without a significant overall learning-gain advantage

A mathematics chatbot can make a lesson more interesting without producing a clear improvement in learning during a brief classroom sequence. Cilia Ricarda Rücker and Sebastian Becker-Genschow examined this distinction through ADA, a domain-specific assistant designed for secondary mathematics. Their study focused on the Heron method for approximating square roots and compared the chatbot with conventional differentiated materials. It offers a useful case of evaluating both the experience of learning and the knowledge students demonstrate, rather than treating enthusiasm as sufficient evidence of effectiveness.

The study involved 195 ninth-grade pupils from German secondary schools, with 102 in the chatbot condition and 93 in the comparison condition. Existing classes were allocated through cluster randomization. The classroom sequence used one period for the pre-test and a double period for the intervention and immediate post-test. The groups differed in prior knowledge despite the allocation procedure, so the analysis examined changes over time rather than relying only on post-test averages. This baseline imbalance is relevant to how confidently any difference can be attributed to the intervention.

ADA was configured around curricular content and pedagogical support. The study describes curated knowledge, retrieval and prompting intended to make responses more relevant to the topic. Pupils could seek help during the mathematical activity, while the comparison group received conventional materials for differentiation. Such a design concerns a tailored assistant embedded in a lesson, not unrestricted use of a general chatbot. The learning tasks progressed from recalling the method to applying it and considering how it might be optimized.

Pupils using ADA reported significantly greater situational interest and generally favorable judgments of usability and usefulness. The broader outcome pattern was more cautious. Differences in reported emotions and cognitive load did not reach significance. Although descriptive learning gains favored the chatbot group, the overall group-by-time interaction for performance was not significant, with a reported p value of .094. The findings therefore provide clearer evidence for immediate motivational engagement than for an overall achievement advantage. A favorable trend should remain a trend rather than be promoted into a confirmed effect.

The study was limited to brief exposure, one mathematical topic and a particular age group. It did not assess delayed retention or repeated use, and novelty could have contributed to interest. Detailed chat processes were not the main object of analysis, leaving uncertainty about which interactions helped or distracted learners. The paper also acknowledges that retrieval and tailored prompts cannot eliminate misleading output. These limits matter because perceived competence of an assistant may increase willingness to use it even when the learner is not well placed to judge its mathematical reliability.

For AIEDHK, a useful follow-up would compare interest, explanation quality and independent problem solving across several lessons. Teachers could inspect a sample of conversations for misleading hints and ask pupils to justify each step of a new calculation without the assistant. Interface design should also reduce the effort of switching between paper tasks and digital support. These are proposed extensions beyond the study. The practical contribution is a measured separation of engagement from achievement: a promising learning experience still needs evidence that students can retain, explain and apply the mathematics themselves.