automated evaluation

A guarded LLM tutor reached its withholding targets in scripted tests, but student learning remains unmeasured

Yusuf Pisan

arXiv preprint

Pisan reports a deployed programming tutor that places an eight-rung help ceiling outside the generating LLM, strips solution code deterministically and judges risky replies against a per-turn contract. Across roughly two dozen scripted turns per calibration run, earnest-reply revisions fell from 43% to 0% and audited ceiling compliance rose from about 77% to 100% after measurement and policy defects were repaired. The evaluation used synthetic personas, not students, so it establishes contract compliance rather than durable learning.