Improving Generative AI Student Feedback: Direct Preference Optimization with Teachers in the Loop

Helpful feedback depends on the course as well as the student's answer. Terminology, the level of explanation, and what an instructor expects a learner to attempt next all shape an appropriate response. An Educational Data Mining study explores whether a language model can learn those preferences from teaching assistants' ordinary grading decisions, while leaving the final feedback under human control.

Juliette Woodrow, Chris Piech, and Sanmi Koyejo developed a workflow in a university probability course. For each student response, teaching assistants saw two AI-generated feedback suggestions. They could choose one, edit it, or write a new comment. Those choices became training examples for direct preference optimization, a method that increases the likelihood of responses people prefer. The updated model then supplied suggestions for a later assignment. The workflow concerned feedback text, not automatic assignment of the student's grade.

The implementation used a Llama 3.1 model adapted to the course, with GPT-4o helping identify relevant observations about the submitted work. Staff reviewed all generated feedback before it reached students. This distinction is central to the contribution: the system learned from actual instructional choices and operated within an existing review process. It was not evaluated as an autonomous tutor communicating unchecked comments to a class.

A controlled evaluation involved ten current or former course staff. They compared feedback on anonymized submissions under blind, randomized presentation, without being told which models had produced it. The adapted model was preferred in 56.8 percent of comparisons, versus 40.2 percent for GPT-4o; in the remaining three percent, neither response was considered suitable. Evaluators also gave the adapted feedback higher insightfulness ratings. These findings concern expert judgments of feedback quality and fit, rather than student achievement.

The classroom deployment supplied a useful counterbalance to the controlled comparison. Across two course offerings with nearly three hundred students and more than ten teaching assistants per term, staff often wrote their own feedback. In one later deployment round, teacher-written comments accounted for 58.7 percent of responses, compared with 27.5 percent selected from the adapted model and 13.8 percent from GPT-4o. A model can outperform another model while still being only one source of useful comments in practice.

The researchers also reported an implementation failure: a formatting bug prevented models from seeing equations in one deployment round, which also had fewer problems contributing preference data. That episode underscores how interface and data handling can change feedback quality independently of model capability. Automated critic scores supplied additional evaluation, but such judgments remain proxies that need to be compared with human review and real student use.

The study did not assess whether learners acted on the comments, improved their revisions, or retained more understanding. Its findings come from one course, and an exploratory fairness analysis was too limited to establish equitable performance across learners. For teaching teams, the practical contribution is an adaptable review-and-learning workflow: retain the ability to reject suggestions, learn from expert edits, and track why comments are chosen. The next question is whether feedback that better matches staff preferences also supports better independent learning.