graphpad prism tutorial statistical significance on graph

Since these simulations were created for values where the true mean in both populations was identical, any declaration of 'statistical significance' is a Type I error. In all three cases, we would have declared the results to be 'statistically significant'. We would have stopped the red experiment when N=61, and the blue experiment when N=92. We would have stopped when N=7 in the green experiment, so would never have seen the dotted parts of its curve. If we followed the sequential approach, we would have declared the results in all three experiments to be 'statistically significant'. It has a P value less than 0.05 when N=92 to N=100. It reached a P value less than 0.05 when N=61 and also when N=88 or 89. The red curve shows the second simulated experiment. It reached a P value less than 0.05 when N=7, but the P value is higher than 0.05 for all other sample sizes. The green curve shows the results of the first simulated set of experiments. The greenish shaded area at the bottom of the graph shows P values less than 0.05, so deemed 'statistically significant'.

The graph plots P value on the Y axis vs. So any 'statistically significant' result we obtain must be a coincidence - a Type I error. These simulations were done comparing two groups with identical population means. Then we repeated the entire simulation three times. We repeated this until N=100 in each group. Then we added one subject to each group (so N=6) and recomputed the t test and P value. We picked N=5 in each group and computed an unpaired t test and recorded the P value. Both groups were simulated using exactly the same distribution. We simulated data by drawing values from a Gaussian distribution (mean=40, SD=15, but these values are arbitrary).

The graph below illustrates this point via simulation.

Simulation to show the dangers of not choosing sample size in advance

The consequence is that the chance of obtaining a 'significant' result if the null hypothesis were true is a lot higher than 5%. The problem with this approach is that you'll keep going if you don't like the result, but stop if you do like the result. If your results are statistically significant result, then stop the study and don't waste time or money on more data collection. Why not do the analyses as you collect data? If your results are not statistically significant, then collect some more data, and reanalyze. To many, calculating sample size before the study starts seems like a nuisance. The appeal of choosing sample size as you go

Graphpad prism tutorial statistical significance on graph