First of all, iím sorry about my bad English and poor understanding of statistics. Iíve just started to learn about both of them.
Iím a novice experimental Psychologist, and I have some problems with the data analyses in one of my experiements. I thik youíll be able to help me.
My experiment has 3 groups that differ in their training phase (G1, G2, and G3). After the training phase, every participant needs to answer three questions in a 0-100 scale. These are the response variables. The 3 questions are presented in a counter-balanced order. There is a Causal question (J1), a Prediction question (J2), and a Preparation Question (J3).
My theoretical hypothesis is a quite complex interaction where:
Answers to J1 (Causal): G1 = G2; G2 > G3.
Answers to J2 (Prediction): G1 < G2; G2 = G3.
Answers to J3 (Preparation): G1 < G2; G2 = G3.
Well, I know that the design may not be perfect (Iím still learning).
The mean judgments are depicted in the following table:
Answers to J1 (Causal question):
G1: 40,19 (M.S.E.=5,38)
G2: 60,20 (M.S.E.=6,39)
G3: 11,29 (M.S.E.=4,32)
Answers to J2 (Prediction question):
G1: 42,30 (M.S.E.=5,47)
G2: 80,00 (M.S.E.=5,09)
G3: 67,59 (M.S.E.=7,93)
Answers to J3 (Preparation question):
G1: 41,34 (M.S.E.=6,36)
G2: 74,58 (M.S.E.=5,66)
G3: 49,81 (M.S.E.=8,26)
(In the graphic, error bars represent M.S.E.)
I think that, in these circumstances, an ANOVA wonít provide any relevant information about the hypotesis (correct?). Because of that, I planned some a priori contrasts.
I decided to use the Dunn-Bonferroni procedure (only because Iíve noticed that is widely used, not because of technical reasons), but you may recommend better alternatives.
Iíve conducted 6 contrasts, all of them between-groups. Then, Iíve corrected the alpha level from ,05 to ,008 with Bonferroni procedure (k = 6 contrasts).
You can check the results here:
G1 vs. G2:
Answers to J1 (Causal question): N.S. (p=0,09)
Answers to J2 (Prediction question): tí(74)=3,16; p<0,0083 (p=0,00205)
Answers to J3 (Preparation question): tí(74)=2,79; p<0,0083 (p=0,0063)
G2 vs. G3:
Answers to J1 (Causal question): tí(74)=4,14; p<0,0083 (p=0,00007)
Answers to J2 (Prediction question): N.S. (p=0,295)
Answers to J3 (Preparation question): N.S. (p=0,038), but p<,05.
The results fit my above mentioned prediction, and we have also found an unexpected tendency with p=.038 (answers to J3, G2 vs. G3) that did not reach the new corrected signification criterion but was below the regular significance level p=,05.
After my supervisors had seen these results, I have some questions:
1. I was told that, in a design like this, always is required an ANOVA prior to any other analysis. In this particular experiment, however, I thik that such an ANOVA wouldnít provide any interesting information about my hypothesis. Post-hoc analyses would be neccesary in any case, so why donít we go directly to the contrasts?
Am I correct?
2. I was also told that, after the ANOVA, the same 6 comparisons can be conducted with a common t test (instead the Dunn-Bonferroni test that I have used). Is it really correct?
3. My correction of alpha with Bonferroni procedure has also been critisiced. I thought that it was the most appropiate since I have multiple comparisons. Again, am I correct?
4. The main argument for not using Bonferroni (or any other correction) is that, according my colleagues, such a correction works ďin my favourĒ this time, since it avoids some ďbigĒ (in absolute valor) and unexpected differences to reach significance level, as my hypothesis predicted.
I would answer that the correction of alpha does not affect the statistical power, only the critical value for rejecting the null hypothesis, and this is valid also when the significant difference is expected. Am I wrong?
Well, I think that thatís all I need for me to start with this experiment and to begin improving my skills. Thanks for your help.