LLM-curated lecture clips offered source-grounded computing help, but the pilot measured use rather than learning

Owen Tang, Alexandra Vassar and Jake Renzella investigate a constrained alternative to open-ended AI tutoring: retrieving short, relevant sections from an instructor's own lecture videos. Their August 2026 preprint describes a system for a large introductory C-programming course at UNSW. Instead of generating an answer, the language model receives lecture transcripts with timestamps and returns candidate intervals. Students can then watch the lecturer's explanation in its original course context.

The evaluation used five questions aligned with different levels of Bloom's taxonomy. A course lecturer independently selected reference segments, and the authors compared three frontier language models with those selections. GPT-5.4 Pro achieved 0.78 recall, 0.53 precision and 0.31 intersection over union. Gemini 3.1 Pro reached 0.60 recall, 0.23 precision and 0.21 intersection over union. Qwen3.5 397B performed substantially worse, with 0.11 recall, 0.08 precision and 0.05 intersection over union. Model-based judges rated the GPT and Gemini selections highly relevant, at 4.87 out of five, and gave sufficiency scores of 4.07 and 4.20.

Those numbers describe overlap and judged usefulness, not student learning. The lecturer selected no segment for two questions because the lectures did not directly cover the requested material. The models nevertheless returned the closest available content. One result was considered adequate, while a memory-leak selection was relevant but insufficient. This boundary matters: a retrieval system may produce a plausible clip even when the source collection cannot fully answer the question.

The authors also report a seven-week classroom pilot using Gemini 2.5 Pro with 903 enrolled students. The system received 343 prompts; 58 percent of participating students returned for another use. The median recommended clip lasted 91 seconds. Common topics included arrays, linked lists and pointers. Students submitted only 21 explicit ratings, with 19 approvals and two disapprovals. Positively rated recommendations were watched for an average of 46 seconds, compared with 126 seconds across all recommendations. These traces show that some learners used and revisited the tool, but they do not reveal whether students understood more, completed work more independently or performed better.

Several limitations narrow the conclusions. The controlled test used only five queries and one lecturer's selections. LLM judges supplied relevance and sufficiency ratings, transcript-only retrieval ignored visual explanations and the deployed pilot used a different model from the strongest evaluated system. The study reports no learner demographics, comparison group, achievement measure or analysis of who benefited. Engagement measures can also be ambiguous: a short watch may reflect efficiency, abandonment or prior knowledge.

For AIEDHK, the design is promising because it keeps assistance grounded in teacher-approved materials and returns learners to a human explanation. A stronger evaluation would preregister a larger question set, include multiple human raters, log when the corpus lacks an answer and compare learning, transfer and unaided performance against ordinary search or generated tutoring. It should also test whether findings generalize beyond one introductory course. The appropriate current claim is feasibility: models can locate useful lecture moments, but educational effectiveness remains untested.