AI lesson agents adapted detectably to learner personas, but delivery lagged content and an LLM judge misranked the leaders

Lin and colleagues ask whether an AI agent that generates a complete lesson can also transform subject matter for a particular learner. They frame that capacity as pedagogical content knowledge: representing a concept in a form that fits a learner's age, prior knowledge, attention and gaps. The first Teaching Monster Challenge turns this idea into an end-to-end instructional-video benchmark. Each system receives a course requirement and free-text learner persona and must create one complete video without human intervention.

The benchmark covers Advanced Placement-aligned Physics, Biology, Computer Science and Mathematics at secondary level. Some items hold the topic constant while changing the learner persona, making adaptation observable. Systems were evaluated on content accuracy, pedagogical logic, learner adaptability, and engagement and multimodal presentation. An automated multimodal LLM judge screened submissions, crowd raters compared shortlisted systems blindly, and teachers, school leaders and university professors formed the final expert panel.

The challenge operated at substantial scale. Forty-six teams generated 1,696 warm-up videos. In the preliminary phase, 77 teams generated 1,612 videos for 32 released items. A 22-item automated rubric advanced ten teams; 59 Prolific raters then contributed 246 pairwise comparisons. The top three systems generated 48 videos for 16 final items. Ten experts reviewed the finalists, with each subject assigned two subject specialists and one pedagogy specialist. Award candidates also submitted reproducible system images to verify the no-human-in-the-loop pipeline.

Content accuracy and pedagogical logic scored higher than engagement and learner adaptability. The authors recorded 6,699 negative flags: 39 percent visual delivery, 27 percent learner adaptation, 16 percent content, 8 percent narration and 10 percent count-based explanation errors. Ineffective visual representation appeared in one third of videos; missing scaffolding in 29 percent, jargon overload in 23 percent and prerequisite gaps in 22 percent. About 17 percent received a critical-fact-error flag. These automated flags are not learning outcomes, but they show how a polished lesson can remain overloaded or mismatched.

A separate study tested whether adaptation was perceptible. Sixty-nine Prolific raters contributed 214 ratings after watching a video and choosing its intended learner from three candidates. A persona-independent retrieval baseline stayed at chance, while shortlisted AI systems were identified above it. The human-video comparison was identified more often than the AI group, although the difference was not statistically significant. The agents therefore adapted in detectable ways, but their broader scores suggest that adaptation was often incomplete.

The benchmark also exposes an evaluation problem. Among the ten shortlisted systems, the automated judge's ranking agreed poorly with the crowd ranking: Spearman's rho was -0.17. Scores clustered near the top of the five-point scale. Repeated automated ratings were reasonably stable, so the issue was not merely rerun randomness; the rubric separated a weak tail but did not distinguish leaders as people did. The authors therefore treat automated screening and human comparative judgment as complementary stages.

The boundaries are important. The study evaluates English-language, AP-aligned STEM videos rather than interactive tutoring, classroom implementation or measured learning gains. The final panel mainly reflects Taiwan's education system, and fixed topics, time limits and 2026 systems constrain generalization. The automated judge's flags are not ground truth. For Hong Kong education, the benchmark works best as an audit template: hold a topic constant, vary a bilingual learner profile, inspect whether examples, pacing, prerequisites and visuals truly change, then measure unassisted understanding and transfer with real learners.