Stating the data-science step increased acceptance of AI code suggestions, while alternatives alone did not

An AI coding assistant can produce valid-looking code while misunderstanding the analytical step a researcher is trying to complete. Dhivyabharathi Ramasamy, Cristina Sarasua and Abraham Bernstein examined whether making that step explicit helps, and whether offering alternative code paths adds value. Their study used CATSci, a notebook interface that lets users inspect, edit, accept or reject recommendations. For education researchers and students learning analysis, the underlying question is how an interface can support deliberate analytical choices rather than make code insertion the default measure of success.

The experiment recruited 28 data scientists from Upwork, each reporting at least one year of relevant experience. Participants were randomly allocated to four groups of seven, combining the presence or absence of workflow-step information with the presence or absence of alternative recommendations. All used a GPT-4-based assistant in Jupyter notebooks. They completed a descriptive task involving correlations in a prepared public dataset and a predictive task involving a financial classification dataset. Task order was randomized, and interaction logs were combined with surveys and written feedback.

The interface allowed prompts to include previous notebook cells, additional instructions and an explicit data-science step, such as exploration or modeling. In the alternatives condition, the assistant supplied a main suggestion plus three other possibilities. Users could examine the code before inserting it. This design made the research question narrower than whether AI helps data analysis in general: every group had AI support, and the experiment varied particular features of the assistance and its presentation.

Providing the workflow step significantly increased the probability that users accepted a recommendation. Offering alternatives alone did not produce a significant difference. Acceptance and task scores also differed between descriptive and predictive work, with the predictive task performing more favorably. Participants often wanted additional instructions and contextual information to obtain useful suggestions. These results suggest that communicating analytical intent may matter more than simply multiplying proposed solutions. They also show that success on one kind of data task should not be assumed to transfer to another.

The findings have practical limits. Seven participants per group provide limited statistical power, and the tasks were intentionally simple. The study's reported power analysis was only 0.5 for a large effect under its assumptions. Users were encouraged to rely on the assistant as much as possible, and the interface did not retain a complete conversational history automatically. Most importantly, acceptance is a behavior, not a guarantee that a method is justified or a conclusion is valid. The results do not establish educational learning gains, scientific reproducibility or superiority over working without an assistant.

For AIEDHK, research-methods teaching could use the workflow idea while retaining independent checks. Before requesting code, a learner could state the question, the current analytical step, the data assumptions and the expected output. After receiving a suggestion, the learner could explain why the method fits, inspect errors and compare a plausible alternative where it matters. A later unaided task could test understanding of the analysis. These are proposed extensions beyond the study. The useful lesson is to make analytical purpose explicit and inspect the result, while keeping convenience, code acceptance and evidential quality as separate outcomes.