Preprint: Raising a tutor metric lowered human ratings in an optimization experiment

An October 8 preprint by Daniel Domínguez Figaredo and Rafael Fernández De la Cruz audited a tutor across 2,000 synthetic learner scenarios and fine-tuned an open model on weak cases. For corrected cases, its adaptivity metric rose from 0.05 to 0.42 while ratings from 31 graduate-student raters fell from 4.46 to 3.03. Repetitive activity choices contributed to the discrepancy. Human ratings were in-sample, and changed prose also complicates interpretation. The study did not test student learning; its open model was a proxy for the original tutor.

For AIEDHK, the result suggests a practical review exercise before a school relies on a single quality score. The following activity is an editorial proposal. A teaching team could take two short lesson sequences and ask what a score would reward in each. One sequence might repeatedly offer a useful explanation. Another might begin with explanation, move to practice and then ask the learner to justify a new example. Reviewers would discuss whether the scoring rule can distinguish those purposes.

The exercise could use a familiar topic such as interpreting a bar chart. At the start, a learner might need help identifying the axes. Later, the learner might need to compare quantities or challenge an unsupported conclusion. The team could write down the evidence that would justify each transition. It would then inspect whether a highly rated sequence actually responds to those changes or simply repeats an initially reasonable move. This makes the intended progression available for discussion.

Students in a teacher-education course could extend the exercise by inventing a response that earns a good score while failing their lesson objective. They would explain the mismatch and propose one observation that the score misses. Another group could challenge that proposal with a different learner example. The purpose would be to improve the explanation of instructional quality, rather than to collect increasingly elaborate numbers that nobody can interpret in a teaching situation.

A local pilot could preserve separate evidence about individual responses, the sequence as a whole and subsequent learner work. Teachers might review whether the assistance becomes less necessary, whether an error recurs and whether the learner can explain a method independently. They could also note cases where repeating an explanation is appropriate. Variety alone would be an inadequate target if it distracts from the learner's needs. Each judgment should have an explicit educational reason.

For development teams, a release discussion could include one successful example and one unresolved case. Reviewers would inspect the complete exchange, not only the aggregate result, and decide what additional check is needed. A change that improves one indicator could remain provisional until other relevant observations have been considered. That arrangement would help prevent the easiest number to improve from becoming the only definition of success.

The proposed approach also makes disagreement useful. If two educators prefer different sequences, they can identify the learner evidence or lesson objective behind their choices. Recording that difference may be more informative than averaging it away immediately. An institution can then decide whether the disagreement reflects unclear criteria, different teaching aims or a case that needs direct classroom investigation before wider adoption.