Lecture-video curation grounded AI help in course material, but the pilot measured engagement rather than learning

Owen Tang, Alexandra Vassar and Jake Renzella examine whether large language models can help novice programmers without generating a fresh answer. Their August 2026 preprint uses LLMs to retrieve short segments from recorded lectures in an introductory C course. The design aims to keep help inside educator-delivered material, reducing hallucination and complexity misalignment while sparing learners from searching multi-hour recordings.

The pipeline begins with Whisper transcripts divided into timestamped partitions. GPT 5.4 Pro and Gemini 3.1 Pro use their native file-search tools; Qwen3.5 397B uses a custom retrieval system combining BM25 keyword search and FAISS semantic search. The models return lecture intervals rather than an open-ended explanation. Five test questions cover factual, conceptual and procedural demands from Bloom's revised taxonomy, including pointer syntax, linked-list memory allocation and debugging a memory leak.

One course lecturer independently selected comparison intervals. The authors measured overlap, precision and recall, then asked Claude Sonnet 4.6, Gemini 3.1 Pro and Mistral Large to judge relevance, sufficiency, redundancy and extraneous material. A score counted only when the three judges were within one point. This creates a useful multi-judge check, but it is still automated evaluation rather than validation by learners or several educators.

The proprietary models retrieved much of the lecturer's target material while casting a wider net. GPT averaged 0.78 recall, 0.53 precision and 0.31 intersection-over-union; Gemini averaged 0.60, 0.23 and 0.21. Both scored 4.87 out of 5 for relevance and just above 4 for sufficiency. Qwen's current configuration performed much worse on overlap and sufficiency. The models sometimes returned nearby foundations when the lecturer found no clip that directly answered a question. That can orient a learner, but it can also make a partial answer look complete.

The deployment used Gemini 2.5 Pro as a supplementary tool in a ten-week C course with 903 enrolled students. During seven weeks it processed 343 prompts, and 58 percent of users returned. Clips had a median duration of 91 seconds. Voluntary feedback included 19 approvals and two disapprovals. Students most often searched foundational topics: arrays appeared in 48 queries, linked lists in 45 and pointers in 40.

The evidence does not yet establish a learning benefit. The benchmark used only five questions and one lecturer. LLM judges could not see on-screen code, transcript errors may affect retrieval and the pilot tracked use rather than demographics, a comparison group, independent performance or learning gains. The authors explicitly call for human expert review and longitudinal outcome research.

For AIEDHK, the study's strongest idea is architectural restraint: AI can route learners to approved explanations instead of replacing those explanations. A school pilot should add teacher-reviewed clips, visible no-answer states, learner ratings tied to actual segments, and unaided pre/post assessments. The system should be judged not only by whether students click and return, but by whether they later solve and explain programming problems independently. That outcome evidence remains the decisive safeguard for adoption.