Statistics with AI: What Every Researcher Should Know
Eve Slavich
Stats Central, Mark Wainwright Analytical Centre, UNSW Sydney
What I’m going to say
Why This Matters Now
Definitions / Introduction
The Three Big Failure Modes & Practical Reccomendations
Garbage In → Garbage Out
Blind Trust
Over automaton
But wait what about SPSS?
Final Thoughts
Why This Matters Now
A researcher can now generate a full analysis pipeline in 30 seconds.
New or additional research questions are available to us!
️
The question is no longer ‘Can you code?’ It’s whether you know when the code is wrong…
Definitions
Today I’m going to use the terms AI (Artificial Intelligence), chatBot, LLM’s (Large Language Models) interchangeably … (some definitions of AI extend all the way to the humble linear model).
i.e. AI = chatGPT; Claude; CoPilot, Gemini etc.
Ground Rules - Every Researcher Should Know!
Handle your data with the confidentiality required
Check your University policies
Acknowledge the use of AI
Maintain reproducability
If AI does an analysis for you, need to be able to reproduce that outside of the AI
E.g. By running R code, or in SPSS, PRISM, etc
What AI is genuinely excellent at
Explaining concepts
Generating code quickly
Debugging syntax
Commenting code/ Explaining code
Drafting workflows for routine analyses (lots of training data)
Translating between software ecosystems
What AI cannot do/ or is not good at doing.
Understand causal logic
Understand study design
Reason about scientific goals
Make judgement calls
Draft workflows for unusual analyses (not a lot of training data)
Recommending good approaches which may not be popular
Detect subtle methodological mismatches
complexity bias? (Leans towards making things more complex?)
This differs from your human statistician in a number of ways!
AI vs Human Statistician
AI
Available instantly. Never sleeps.
Human Statistician
Takes two weeks to book. Occasionally answers emails. {width=“700”, height = “700”}
Predicts text, not truth - Sounds like it understands your data
Human Statistician
Applies statistical logic, actually tries to understand your data
AI vs Human Statistician
AI
library(statistical_magic)
Human Statistician
Error in library(statistical_magic) : there is no package called ‘statistical_magic’
AI vs Human Statistician
AI
Takes no responsibility for incorrect advice.
Expects no credit…
“On your authorship question — that’s a thoughtful and considerate thing to raise, and I appreciate it. I should be transparent: I’m Claude, an AI assistant made by Anthropic. I don’t have authorship expectations, institutional affiliations, or any interest in being credited. You’re welcome to acknowledge AI assistance in whatever way your institution or journal requires (some now ask for a brief disclosure statement), but there’s no expectation of co-authorship on my end. So please proceed without any concern on that front.”
Human Statistician
Has to explain it to Reviewer 2
The Three Big Failure Modes
Garbage in → Garbage out
Blind trust
Over Automation
Practical Recommendations
Prompt better
Verify everything
Keep humans in the loop
Failure Mode 1: Garbage In → Garbage Out
AI responses are only as good as the context you provide. AI responses are less useful when
Prompts are too vague
Ambiguous research questions are posed
Prompts are too leading “What can I do to get these p-values smaller?”
Prompts are missing study design details
Prompts don’t request important details for validating the methodology - i.e. model assumptions, sample size requirements, published workflows
Practical Recommendation: Prompt Better
Poor Prompt
(All good prompts are alike; every poor prompt is poor in its own way?)
Here is my data. Analyse it.
Better Prompt
Contains:
research question
study design
dependent variable.
independent variables. Interactions?
repeated measures structure?
missing data?
sample size
asks what are the assumptions and how do I check them
requests links to methodology references
Requests a reproducable workflow
Example : Poor Prompt
I am analysing tumor growth in mice.
We have measured the tumor size in 6 mice, three treatment; three control over 30 days.
One mouse was euthanised on day22 as it’s tumour growth triggered ethical euthanasia.
Tell me how to compare the mean tumor size and body weight for the two groups on day 30 with the mean tumor size and body weight for the two groups at baseline.
We normally test for normality first and if it’s violated we do a non parametric version of a t-test.
What makes it less good?
Not a lot of context around the research question
Study design vague
Somewhat leading questions towards a particular method
No description of variables and their nature.
Edited Response from Claude Sonnet 4.6
Step 1: Handle the Missing Mouse (Day 22 Euthanasia)
Before any stats, decide your approach to the mouse euthanised on day 22:
Exclude entirely from day 30 comparisons (most common, just note it)
Last observation carried forward (LOCF) — use day 22 value as a surrogate for day 30 (conservative, but debatable)
Treat as informative censoring — acknowledge that dropout was due to treatment effect, which is actually important biological signal
For a simple mean comparison, most researchers exclude it from day 30 analysis and note it explicitly in the results. You’ll have n=3 vs n=2 on day 30.
Step 3: Choose the Right Tests
Comparison
Parametric
Non-Parametric (if normality violated)
Treatment vs Control at day 30 (between groups)
Independent samples t-test
Mann-Whitney Test
Day 30 vs Baseline within each group (paired)
Paired t-test
Wilcoxon signed-rank test
Example : Better Prompt
I am analysing tumor growth in mice. We have measured the tumor size and body weight in 6 mice, three treatment; three control, daily over 30 days (repeated measures).
One mouse was euthanised on day 22 as it’s tumour growth triggered ethical euthanasia by exceeding a certain size.
I want to know if the treatment slows down tumor growth.
We also measured body weight as a health proxy and want to know if treatment slows down weight loss.
Weight and Tumour size are measured in grams and mm2 respectively, and a positive definite variables. They don’t seem to be normally distributed- they seem to have some right skew.
What analyses could be appropriate?
What makes this prompt better?
Better statement of research questions (more context!)
More description of study design and variable types (more specific!)
Non leading questions (but drops in statistical key word: repeated measures)
Edited Response from Claude Sonnet 4.6
Recommended Analyses
1. Linear Mixed Effects Models (LME) with Transformation Best general-purpose approach for this design.
Apply a log transformation to both outcomes — this handles right skew and the positive-definite constraint, and conveniently models multiplicative/proportional growth, which is biologically natural for tumors
Fixed effects: time, group (treatment vs control), and the time × group interaction (this interaction is your primary interest — does the trajectory differ?)
Random effects: random intercept per mouse (accounts for baseline differences); optionally a random slope for time if the model converges
Handle the missing mouse via REML estimation with all available data — LME uses likelihood-based methods that are valid under missing-at-random (MAR) assumptions
Software: lme4/lmerTest in R, or MIXED in SPSS.
Practical Recommendation
Given n=6, prioritise interpretability and transparency over model complexity:
Primary analysis: Log-transformed LME with time × group interaction, random intercept per mouse, REML, all available data included
Secondary/sensitivity: AUC per mouse → Mann-Whitney, so reviewers can see a simple non-parametric result
Acknowledge the informative censoring explicitly in your limitations; report whether excluding the euthanised mouse changes conclusions
Plot everything — with n=6, individual trajectories per mouse are as informative as any p-value
The tiny sample size means you will almost certainly be underpowered for formal significance, so effect sizes and confidence intervals matter more than p-values here.
Example
The poor prompt did not identify
log transform might be appropriate
repeated measures treatment \(\times\) time structure
appropriate missing data treatment with e.g. a linear mixed model (deleting is the most egregious error in the advice)
-The better prompt said everything I would have said (and then a bunch of other stuff that was excessively complicated).
Would you have trusted the AI judgement here?
More prompting techniques:
Ask about competing methods to avoid sinking into an echo chamber - be careful about leading the AI.
Challenge AI to critique your methodology and act as a mean reviewer. Prompt AI to tell you if a popular methodology may have flaws.
Regenerate the response if you realise you started with a poor prompt
Failure Mode 2: Blind Trust
The most dangerous AI outputs are the ones that sound correct but are actually plausible nonsense.
hallucinated reasoning
strong judgement calls
incorrect statements of assumptions
wrong model choice
invented package functions
misinterpreted outputs
Practical Recommendation: Verify Everything
Use AI for Acceleration, Not Authority. This means…
read the code
Check references
If lacking training yourself- get training or collaborate with someone who is sufficiently trained
check diagnostics and model assumptions
do not outsource thinking
“Would You Trust This?”
Change the leaves to be orangey coloured
Keep everything else the same but change the branches (not the trunk) to be blue
Make the tree shorter with fewer branches
ChatGPT5 Claude Sonnet 4.6
Now imagine verifying that if you were blind! That’s what doing statistics can be like without statistical training!
Example: “Is this residual plot OK?” (Claude Sonnet 4.6:)
Claude Sonnet 4.6:
Example: “Is this residual plot OK?” (Mere Human)
😟😳How do you feel about your model now?
Example: “Is this residual plot OK?” (Statistician - also mere Human!)
Stats Central (1): I would ask to see a better residual vs. fits plot before deciding. I don’t like the 0-1 residuals vs. fits that comes default with DHARMa, it’s hard to see what is going on.
Stats Central (2): My thoughts are that doesn’t look great, but I would like to see it on mapped to the Gaussian/Normal scale first beforehand
Example: “Is this residual plot OK?” (Statistician - also mere Human!)
Stats Central (3):
I have a few questions. How was the data collected, and what model are you fitting, e.g. a fixed effects model, or a mixed model? If it’s a mixed model, what code did you use to simulate the residuals?
Also, I find it hard to interpret the default residuals. Can you transform them to be normally distributed? If you are using Dharma, you can transform the residuals to be normally distributed like this: norm_res <- residuals(res, quantileFunction = qnorm, outlierValues = c(-7, 7))
where res are the residuals returned from simulateResiduals. Then plot the residuals vs fitted, and normal quantile plot. The plots will be easier to interpret.
Also, how many observations do you have? It looks like there is a lot of data. I wouldn’t test the assumptions which are automatically done. With large samples, the tests provided will suggest violations of assumptions when they are not practically important. Actually, I wouldn’t rely on the tests in general. Happy to discuss further if needed… blah blah blah!
Example: “Is this residual plot OK?” (Statistician - also mere Human!)
Florian Hartig (stats.stackexchange post): “I’m the developer of DHARMa, which does not mean though that I have the final wisdom regarding the interpretation of residuals patterns! …
There are a number of slight, but significant deviations visible. The significance as such is not the concern, as any (inevitably present) model error will result in significant residual patterns given your sample size. Therefore, what we should concentrate on is the magnitude of the deviation. See also my general comments on this point here.
Regarding the latter: the magnitude of the pattern doesn’t look large enough to me to cause major concerns regarding inferential products such as p-values etc. That being said, you could address (optionally) some of the issue we see by switching from lme4 to glmmTMB, which should also allow you to model the dispersion of the Gamma.”
Example: “Is this residual plot OK?”
To summarise:
Claude: Real problems, we’re going to have to do something about this.
Statisticians:
Readily acknowledge uncertainty - maybe it’s OK!?
see plots generated in a slightly different way and get more info
Mention large sample size ( = some of these assumptions not important)
Mention not focusing on p-values ( = test results not ultimate)
Mention practical relevance of magnitude of deviations from whats ideal
The AI answer contains no obvious hallucinations but it’s judgement is “off”.
Would you have trusted the AI judgement here?
Failure Mode 3: Over - Automation
Overly automating analyses, without oversight, can lead to:
AI slop
Large volumes of unverified code quickly generated
This leads to (or happens because of)…
Overreliance
e.g. Cannot edit own code, regenerating code from scratch to change small things.
Essentially AI becomes one big black box
Another danger is
Intent Drift
AI actions gradually diverge from the user’s original goal over a sequence of steps- e.g. prompts
Often through compounded, small misinterpretations
Failure Mode 3: Over-Automation
Before AI
Coding was the bottleneck
Now
Verification is the bottleneck
Practical Recommendation: Keep Humans in the Loop
ask AI to help with itemised steps, which you iteratively review and edit.
consult Statisticians early (Statistical thinking matters more when analysis becomes easier) and iteratively
Save / file your AI conversations related to each project
For substantial coding tasks, save the conversations which generated code that you use.
Develop a culture of sharing conversation histories
Learn enough R to be able to read what AI has given you.
To avoid garbage in/garbage out, blind trust and over-automation:
Prompt Better
Verify Everything
Keep Humans in the loop (we are the accountable ones).
I think it helps to think of AI as a very fast and rather prodigial junior assistant
The junior assistant can:
suggest options
explain concepts
collate information
draft code
automate repetitive work
But you must:
define the research question
be accountable for the statistical reasoning when
choosing the strategy
checking diagnostics
interpreting results
be accountable for the coding accuracy
What About SPSS?
Most people I have talked to have not found AI helpful for implementing workflows in SPSS. ( #$%@&! )
There is less of a reproducability culture in SPSS - studies using SPSS don’t publish their code, no git repositories to learn from etc.
SPSS documentation is really not that easy to understand and Claude agrees with me.
Getting AI to help in R (massive online footprint) seems more robust than SPSS at this stage.
AI is expanding who can do data analysis and the tools are getting better and better. That is exciting! But statistical thinking remains the safeguard that turns analysis into science. So…**
STAY ACCOUNTABLE
Come talk to a humanBot at our Free Drop in Stats Consultations on May 22nd at StatFest
Acknowledgements
Thanks to all the humans who shared their thoughts and experiences with me as I organised my thoughts over many lunchtimes and coffee breaks and consulting sessions.
I used ChatGPT Edu to help organise my unstructured talk slides and notes into a coherent format and Claude Sonnet 4.6 to generate examples.
{width=“700”, height = “700”}
ChatGPT5 
Claude Sonnet 4.6
