A brief prompt scaffold changed how secondary students used ChatGPT but did not improve knowledge gain

Klar asks why students who find ChatGPT easy to operate may still use only a small part of its learning potential. The mixed-methods experiment involved 106 Grade 9 and 10 students from five German secondary schools. In small after-school sessions, groups were randomly assigned to use a ChatGPT-3.5 research interface with or without additional support. Both groups investigated cognitive biases for 20 minutes. The supported group received about 2.5 minutes of instruction, suggestions for follow-up and adaptation prompts, and controls for response length and language difficulty.

The support clearly changed interaction behaviour. Students in the supported condition used an average of 6.00 adaptation prompts, compared with 2.51 in the comparison condition. The difference was statistically significant with a large reported effect, partial eta-squared of .283. Supported students also asked more total questions, 7.8 compared with 4.8, used more follow-ups and engaged in less off-topic conversation. Chat logs therefore show that a small scaffold can make student dialogue more iterative and tailored.

The learning measures tell a different story. The support did not significantly reduce extraneous cognitive load and did not significantly improve knowledge gain. Both groups reported high usability and satisfaction. That combination is educationally important: liking a chatbot and asking more sophisticated questions do not guarantee that learners understand or retain more. Interaction quality may be a mechanism worth developing, but it remains an intermediate outcome unless linked to independent learning evidence.

The study is deliberately small and bounded. Students completed one unfamiliar topic in a 20-minute session and could not use ordinary web search alongside the chatbot. The sample may be too small to detect modest learning effects. Explicitly telling the supported group that adaptation is helpful could create demand characteristics. The research does not establish long-term self-regulated learning, transfer across subjects or how students use unrestricted tools at home.

For classroom design, the findings argue against treating a list of prompt tips as a complete intervention. Teachers can combine question adaptation with source comparison, retrieval and explanation. For example, a student might request a simpler explanation, then identify a claim to verify, compare it with an assigned source and explain the concept without the chat. A teacher can score both the inquiry process and the unaided explanation.

The interface also matters. Follow-up suggestions can help novices notice options they might not invent, but they can channel every learner into the same pattern or add cognitive clutter. Supports should be optional, age-appropriate and tested with learners who have different language proficiency and accessibility needs.

For Hong Kong schools, bilingual studies could test whether language controls support Cantonese- and English-medium learning without simplifying disciplinary meaning. Longer units should measure delayed retention and transfer. The paper's central lesson is precise: brief prompting support changed what students did with ChatGPT, but this experiment did not show that the behavioural change improved learning.

A future study could vary the scaffold itself, compare optional and mandatory supports, and examine whether students transfer adaptation strategies to a new topic after the interface suggestions disappear. That would test whether behaviour becomes a learner capability.