I have just arrived to the forum and really need some advice. Please consider the following problem:
In a psychological experiment participants are asked to intuitively decide if visual stimuli, a light spot, were produced by device “A”. A series of similar but different stimuli are presented and each time the participant as to decide whether it came from device “A” or not. I say intuitively because they have no real way to know if it came from “A” or some other device, but due to an experimental manipulation that we introduce we believe participants are able to intuitively identify above chance stimuli “A”.
Although participants are not told this, stimuli are produced by only two devices “A” and “B”. For each of these devices there are 10 different, but very similar, light spots. The difference between stimuli within-device type is considerably small, and even between devices is not very large, but there’s always some characteristics that distinguish them (between “A” and “B”). By the end of the experiment, participants vary in the degree of acquired awareness regarding these between-device differences. The vast majority believes there were around 4 different devices (in reality there are only 2). Stimuli “A” and “B” are presented an equal number of times (participants do not know this) and in randomized fashion. So probability of each type of stimuli to occur is 50%
The answer is binomial: yes –it is device “A” or No – it is not device “A”.
Question 1: What statistical test shall we use to evaluate if participants identify stimuli “A” above change?
We thought of :
1 - Signal detection theory
2 – Cumulative Binomial Probability
We think both are correct and equivalent, right?
However, our concern was that results could be explained by a rejection of “B” rather than a selection of “A”, thus we carried out a second experiment.
This experiment is very similar to the first experiment with the exception that now stimuli are produced from 4 different devices “A”, “B”, “C” and “D”. The probability of occurrence of each type of stimuli is the same, i.e. 25%. Again, participants are not told the number of different devices.
The answer is still binomial: yes –it is device “A” or No – it is not device “A”.
Question 2: What statistical test shall we use to evaluate if participants identify stimuli “A” above change? That is, the probability to chose “A” is bigger than the probability to chose the other devices?
Worth of noticing that the devices are different across participants so that the false-alarms of for example “B” cannot be “clustered” across participants and treated as a specific type of false alarm (the same for “C” and “D”). Our emphasis is always on finding out if the identification of “A” is higher than chance, both at participant level and group level