Designing Safe and Relevant Generative Chats for Math Learning in Intelligent Tutoring Systems

An AI mathematics tutor needs to answer more than mathematical questions. Learners may joke, change the subject, or disclose something that calls for a careful response. A Journal of Educational Data Mining article examines these realities through two studies: a structured growth-mindset conversation deployed in the Rori tutor, and an offline evaluation of textbook-supported answers to conceptual mathematics questions. Its contribution is evidence about system behavior, rather than a test of mathematics achievement.

Zachary Levonian and colleagues first developed a conversational design using GPT-3.5, moderation checks on inputs and outputs, and predefined responses for different levels of concern. The design kept the exchange focused on a specific educational purpose. Researchers also used adversarial testing and ongoing review, recognizing that a moderation score alone cannot describe every inappropriate or unhelpful interaction. The work treats the surrounding system as part of the intervention, rather than assuming a model is safe by itself.

A classroom usability phase involved 109 students across six classrooms and 252 conversations. A subsequent field deployment involved 8,168 users, 8,755 conversations, and 126,278 messages. Researchers reported no overtly objectionable generated outputs in their monitoring. That observation is useful, but it is bounded by this conversation format, model, user population, and review process. It cannot establish that unrestricted dialogue or another deployment would behave in the same way.

The more revealing issue concerned responses to student messages. In a review of unflagged conversations with elevated moderation scores, forty included material that warranted caution or correction. The system responded appropriately in thirty-seven, while three responses ignored or equivocated about the problem. A message can therefore avoid obviously prohibited content and still fail to provide an adequate educational response. The authors discuss the need for better handling of sensitive inputs and clearer pathways to human support.

The second study examined retrieval-augmented generation: supplying relevant textbook material when a model answers a question. Researchers selected 51 answerable conceptual or factual mathematics questions from a larger sample of student discussion posts. Nine educators and educational-technology designers compared answers produced with different amounts of guidance. They tended to prefer moderately guided answers over both unguided answers and strongly constrained ones. Greater resemblance to a source document did not automatically make an answer more useful.

This distinction matters for classroom design. A response may closely repeat a textbook yet fail to address the learner's confusion, while a more conversational answer may be preferred without being the strongest teaching choice. The study's preference judgments came from adults, not the students who asked the questions, and the evaluation concerned single-turn answers. It did not establish a relationship between those preferences and subsequent learning.

For developers and educators, the practical lesson is to evaluate several dimensions together: subject accuracy, relevance to the question, handling of student inputs, and the learning activity surrounding the exchange. The observed deployment supports the feasibility of a carefully structured conversation, while its exceptions show why monitoring must examine subtle failures as well as explicit harmful output. Broader claims about safety, student benefit, or instructional effectiveness require additional evidence in the intended setting.