Evidence of impact and interpretational limits of generative AI in STEM education: A systematic review and meta-analysis on cognitive learning outcomes

Claims that generative AI improves learning often rest on a pooled average across very different studies. A systematic review in Artificial Intelligence Review shows how much that average can depend on publication bias and instructional comparisons. The conventional analysis favored AI-supported STEM learning, but an alternative analysis accounting for selective publication placed the overall estimate close to zero. Neither result describes every classroom or every use of AI.

Chiara Boolzen and colleagues searched ERIC, PsycINFO, and Web of Science, with the search updated on 7 May 2026, and added studies through citation tracking. The review included 85 studies involving learner interaction with generative AI and a comparison condition. Of these, 49 studies supplied 59 effect sizes suitable for the quantitative synthesis. The pooled outcomes were externally assessed cognitive performance, a useful boundary that separates demonstrated knowledge or skill from students' impressions of how much they learned.

The initial random-effects analysis produced Hedges' g of 0.839, conventionally interpreted as a large standardized difference. However, between-study variation was extreme: the reported I-squared was 96.32 percent, and the prediction interval ranged from −1.52 to 3.20. A prediction interval describes the spread of plausible effects across comparable future settings, rather than uncertainty around one overall mean. Here, it encompasses negative, negligible, and strongly positive outcomes.

The authors then used robust Bayesian meta-analysis, which combines models with different assumptions about effects, heterogeneity, and publication bias. The adjusted mean was 0.076, with a 95 percent credible interval from −0.37 to 0.79. This weakened the case for a broadly positive average. It should not be read as proof that AI never helps, or as a precise estimate of no effect: the interval remained wide, and considerable unexplained variation persisted after the adjustment.

A central interpretive issue was what learners actually did in the intervention and comparison conditions. The review distinguished designs that changed students' cognitive activity, rather than classifying studies solely by whether AI was present. Differences in learning outcomes and in these instructional arrangements accounted for some variation. Yet the analyses did not establish a reliable combination of features that guaranteed a large benefit. Such comparisons across studies remain vulnerable to other differences between populations, tasks, and implementations.

The evidence base also had important weaknesses. None of the included studies received a low overall risk-of-bias rating under the review's assessment. Learner characteristics and interaction processes were often poorly reported, making it difficult to examine prompt quality, verification behavior, prior knowledge, or the amount of thinking delegated to a system. The review's own model choices, search coverage, coding, and assumptions about publication bias also shape its conclusions.

For educators, the paper offers a way to ask better questions of an attractive headline. What was the comparison activity? Was knowledge assessed independently? Did AI add useful practice or replace the activity students needed to learn? For researchers, it strengthens the case for transparent reporting, comparable instructional conditions, and publication of null results. The contribution is a more cautious interpretation of an uneven literature, with attention to the design of learning rather than a universal verdict about the technology.