← Zurück zu Forschungsnachrichten
Editorial cover of undergraduate learners and a lecturer examining a course-grounded RAG chatbot alongside flat learning and motivation outcome traces
ZeitschriftenartikelPeer-reviewed study202625. Juli 2026· 4 min

AI chatbots in higher education: Comparing expectations to evidence

500-Wörter-Zusammenfassung

Editorial cover of undergraduate learners and a lecturer examining a course-grounded RAG chatbot alongside flat learning and motivation outcome traces

Thoeni and Fryer test a claim that universities increasingly encounter in product demonstrations: if a generative-AI chatbot is grounded in trusted course content, available whenever students need it, and capable of answering questions or generating practice, will it improve learning? Their 2026 open-access article in Computers in Human Behavior Reports reports a semester-long randomized field experiment rather than a satisfaction survey. The result is important precisely because it is negative: the retrieval-augmented generation, or RAG, chatbot did not produce a statistically significant improvement in any of the measured learning-related outcomes.

The experiment took place in three sections of an introductory Principles of Marketing course at a state university in the southeastern United States. One section was asynchronous online, one was a small face-to-face class, and one was a large face-to-face class. After excluding students who dropped or were repeating the course, the study included 454 undergraduates: 231 in the control group and 223 in the treatment group. Students were randomly assigned within each class section after the add/drop period, helping balance the conditions across delivery mode and class size.

The design covered a 16-week semester, with the chatbot intervention operating for 12 calendar weeks. Before treatment, all students completed the same course work and first test. The control group then continued receiving participation credit for study sessions using custom Quizlet flash cards. The treatment group received equivalent credit for at least one chatbot session per chapter, while retaining access to Quizlet without additional credit. Students in both groups could use their assigned support as often as they wished. This made the comparison closer to adding a course chatbot to realistic study options than to replacing instruction.

The chatbot was more carefully constructed than a general-purpose chat window. It ran in a secure university Microsoft Azure environment using Copilot with GPT-4o. The RAG content included instructor-provided concepts, glossaries, learning objectives, and lecture transcripts organized by chapter, but not test questions or textbook content. System instructions gave the tutor goals, a conversational role, and functions for discussion, explanation, multilingual interaction, and short quizzes. The researchers tested its consistency and accuracy and reported that it answered 199 of 200 course test questions correctly, even though those questions were not included in its knowledge base.

The outcomes came from several sources. Students completed pre- and post-treatment measures of individual interest and self-efficacy. Engagement included emotional, participation, performance, and skill scales, weekly self-reports, electronic-book usage, and chatbot session counts. Academic achievement was represented by standardized scores from a common first test before treatment and a common fourth test at the end of the term. The analyses used difference-in-differences models and controlled for gender, age, race, weekly job hours, and whether the course was face-to-face or asynchronous online.

Across interest, self-efficacy, engagement, and test scores, the critical treatment-by-time interactions were not statistically significant. Self-efficacy rose slightly over the semester for students as a whole, and several engagement measures fell, but neither pattern was attributable to chatbot access. For achievement, the treatment-by-time interaction was also non-significant. In other words, the study did not find that adding the course-grounded chatbot changed the measured trajectory relative to the control condition.

Students nevertheless viewed the tool positively. Treatment students reported high enjoyment, perceived help with course material, and interest in having a similar chatbot available in other courses. Their own ratings were more cautious about whether the chatbot improved grades or interest in marketing. This gap is one of the paper's most useful findings: liking an AI tutor, finding it convenient, or wanting continued access does not establish that it improves learning. Adoption and educational effectiveness must be evaluated separately.

The null result also has boundaries. The study involved one introductory subject, one instructor, and three sections at one university. The control group had a legitimate study aid, which sets a more demanding comparison than no support. The chatbot did not remember prior sessions, limiting personalization and social continuity. The researchers could not fully capture how students changed their broader study habits, and the final test covered different course material from the baseline test even though scores were standardized. More advanced learners, other subjects, alternative tutoring strategies, or stronger integration with classroom activity could produce different results.

For AIEDHK, the practical lesson is to evaluate a designed intervention before scaling a platform contract. A course-grounded chatbot may be accurate and popular yet still add no measurable benefit to existing study support. Pilots should define the expected mechanism, compare against a credible alternative, measure independent achievement and engagement over time, inspect actual usage and substitution effects, and test whether memory or personalization helps without creating unacceptable privacy costs. The paper does not show that RAG tutors can never work. It shows that grounding and availability alone are insufficient evidence of learning value.

Verwandte Beiträge

An East Asian female university student and a South Asian male language tutor compare bilingual economics notes and two AI explanations in a sunlit campus language centre
Tool / Datensatz2026
Tool / Datensatz 226

Preprint: Students’ access to generative AI did not ensure equally usable support across languages

Ming Li, Qin Xie, Ariunaa Enkhtur, Lilan Chen, Fei Cheng

arXiv preprint (work in progress)

Interviews with 42 undergraduates at universities in China, Japan, and Mongolia explored how students selected AI tools, chose interaction languages, and checked outputs. Participants described different routes from technical access to useful academic support, including switching languages when responses seemed inadequate. This work-in-progress preprint presents situated accounts rather than national comparisons or measured learning gains. It raises practical questions about evaluating an AI service in the languages students actually use.

preprintmultilingual educationAI access
500-Wörter-Zusammenfassung lesen →
A South Asian female lecturer and a White male university student examine two differently annotated statistics feedback sheets beside a hand-drawn confidence-interval diagram
Zeitschriftenartikel2026
Zeitschriftenartikel 216

Students’ feedback ratings reflected source credibility without consistently matching expert judgments

Tanya Nazaretsky, Hagit Gabbay, Tanja Käser

Computers and Education: Artificial Intelligence

Researchers compared human and AI feedback on 979 responses from six university STEM courses, then examined ratings from 472 students. Expert analysis found broadly similar pedagogical quality, while students’ judgments were associated with their views of the source’s credibility. The study did not measure feedback uptake or learning gains. For educators, it raises a practical question: how can students justify an evaluation of advice before deciding whether to use it?

feedback literacysource credibilityhigher education
500-Wörter-Zusammenfassung lesen →
A student drafts an essay at a window ledge while a peer points to an argument map and a lecturer observes.
Zeitschriftenartikel2026
Zeitschriftenartikel 202

Teacher feedback VS AI-assisted peer feedback in L2 writing: A quasi-experimental study in a Chinese University

Jinfeng Tang, Ping Li, Rong Luo

PLOS One

An eight-week study compared teacher comments with peer feedback supported by DeepSeek in two university English classes. Both approaches supported better revised essays, but their improvement patterns and feedback focus differed. An initial vocabulary-diversity advantage for the AI-peer class did not persist into the second task, and sentence-complexity results were limited. Because intact classes received different teaching packages, the findings cannot separate AI's contribution from peer discussion or other class differences.

Peer LearningWriting EducationAI Feedback
500-Wörter-Zusammenfassung lesen →