Problem: Cumulative binomial/multinomial, signal detection?

Printable View

• Dec 19th 2012, 05:13 AM
PsyCog
Problem: Cumulative binomial/multinomial, signal detection?
Dear all,

I have just arrived to the forum and really need some advice. Please consider the following problem:

In a psychological experiment participants are asked to intuitively decide if visual stimuli, a light spot, were produced by device “A”. A series of similar but different stimuli are presented and each time the participant as to decide whether it came from device “A” or not. I say intuitively because they have no real way to know if it came from “A” or some other device, but due to an experimental manipulation that we introduce we believe participants are able to intuitively identify above chance stimuli “A”.

Experiment 1:

Although participants are not told this, stimuli are produced by only two devices “A” and “B”. For each of these devices there are 10 different, but very similar, light spots. The difference between stimuli within-device type is considerably small, and even between devices is not very large, but there’s always some characteristics that distinguish them (between “A” and “B”). By the end of the experiment, participants vary in the degree of acquired awareness regarding these between-device differences. The vast majority believes there were around 4 different devices (in reality there are only 2). Stimuli “A” and “B” are presented an equal number of times (participants do not know this) and in randomized fashion. So probability of each type of stimuli to occur is 50%
The answer is binomial: yes –it is device “A” or No – it is not device “A”.

Question 1: What statistical test shall we use to evaluate if participants identify stimuli “A” above change?

We thought of :
1 - Signal detection theory
2 – Cumulative Binomial Probability

We think both are correct and equivalent, right?

However, our concern was that results could be explained by a rejection of “B” rather than a selection of “A”, thus we carried out a second experiment.

Experiment 2:

This experiment is very similar to the first experiment with the exception that now stimuli are produced from 4 different devices “A”, “B”, “C” and “D”. The probability of occurrence of each type of stimuli is the same, i.e. 25%. Again, participants are not told the number of different devices.
The answer is still binomial: yes –it is device “A” or No – it is not device “A”.

Question 2: What statistical test shall we use to evaluate if participants identify stimuli “A” above change? That is, the probability to chose “A” is bigger than the probability to chose the other devices?
Worth of noticing that the devices are different across participants so that the false-alarms of for example “B” cannot be “clustered” across participants and treated as a specific type of false alarm (the same for “C” and “D”). Our emphasis is always on finding out if the identification of “A” is higher than chance, both at participant level and group level

Thank you,
PsyCog
• Dec 19th 2012, 03:48 PM
abender
Re: Problem: Cumulative binomial/multinomial, signal detection?
Hi. Welcome to the forums.

Regarding your questions, I first need some clarification. Does the participant know what exactly "A" is? Is the participant shown "A" before the experiment, so he or she can acquire a baseline? Or, is "A" something that common folks understand? Example of this: If "A" is Nike and "B" is any other type of sneaker, and we assume wearing Nike shoes is something everyone has done at some point, then one has a reason to identify "A" or not. Otherwise, a participant is calling out "A" and "other" indiscriminately.

Do you understand what I am trying to ask?

-Andy

EDIT: I believe my question is answered in this line: '... but due to an experimental manipulation that we introduce we believe participants are able to intuitively identify above chance stimuli “A”.' My oversight.
• Dec 19th 2012, 04:20 PM
abender
Re: Problem: Cumulative binomial/multinomial, signal detection?
A few more things to clear up:

Quote:

Question 2: What statistical test shall we use to evaluate if participants identify stimuli “A” above change? That is, the probability to chose “A” is bigger than the probability to chose the other devices?
1) Are we testing for (correct) identification or simply what the participants' choose as their (possibly incorrect) answer?

2) What does 'identify stimuli "A" above change' mean? Do you mean 'chance' instead of 'change'? If not, then are we testing for CORRECT identification of "A" vs. correct identification of B, OR just choosing "A" vs. just choosing "other"?
• Dec 21st 2012, 07:13 AM
PsyCog
Re: Problem: Cumulative binomial/multinomial, signal detection?
Hi Abender,
Thanks for your interest!

Answering to your first thread: Yes, that's exactly what we are trying to test, if participants intuitively choose "A" more often than the others. They haven't seen "A" before, and there is nothing that can objectively (any cognitive process) help them on the choice, but we believe a kind of implicit knowledge would lead them towards choosing "A".

Answers to your second thread:

1) Didn't really understand your question here. I would say that we are testing for correct identification. I believe we are interested in seeing if the relation between “correct hits”, “false alarms” and “misses” is above chance level. The doubt in the second experiment is how to treat the other devices (which bring us to your second question)

2) yes, it was a typo I meant “above chance”.

Our aim would be to test the probability of choosing "A" vs. just choosing "other". The question is how to treat “other”. As I’ve wrote before “B, C and D” are always different across participants, so we cannot treat them as one (at least at the group level). On the other hand, I’m concerned that averaging false identification rates (i.e, false alarm) of “B,C and D” and comparing it with “A” might be misleading. For example, “C” and “D” might be always rejected (i.e, correct rejection) but “B” often chosen (i.e., false alarm). In this case even if “A” was chosen more frequently than the average of “B,C, D” maybe it’s not correct that “A” was identified.

But on the other hand (even if the above happens)… at the group level (all subjects together) if accuracy/sensibility to identify “A” is higher than 25% (use % here to simplify) can’t we say that participants were inclined (above chance level) to chose “A”, and thus they have this implicit knowledge on how “A” should be?

I hope it is clear now my dilemma..
Any comment, suggestion is very welcome.

Thanks