Thread: How can I tell if my trend is real?

Hi guys,

I'm not sure if this should be in advance stats - perhaps it is too basic.

I'm working on a project that looks at the amount of cosmic rays coming out of a particular region of the sky in certain time intervals.

There are 13 readings taken:

9,9,6,6,7,1,6,6,1,6,4,1,0

When this is plotted, it looks like a downward trend - meaning the source is becoming less active - but it's difficult to say with such small statistics.

I though perhaps I should try to quantify the likelihood of getting such numbers if the source was essentially a constant source - ie the trend is just due to fluctuations.

Could someone please suggest how I might think about doing this?

You know, I didn't think there was anything there based on the data you posted, but I fit a model and it actually looks like time is quite significant.

The obvious thing to do is just fit a line to the data, which is what I did. I don't know of any other options, without knowing more about the process involved. If you wait 6 months, I'll have taken a course on longitudinal data and then I'll have a better answer.

Warning: You should only be testing this if you were expecting to find a trend in time BEFORE you did the experiment. It is very bad form to test hypotheses suggested by the data on the same data set.

Originally Posted by Guy

You know, I didn't think there was anything there based on the data you posted, but I fit a model and it actually looks like time is quite significant.

The obvious thing to do is just fit a line to the data, which is what I did. I don't know of any other options, without knowing more about the process involved. If you wait 6 months, I'll have taken a course on longitudinal data and then I'll have a better answer.

Thank you for your response. When you say "fit a line to the data", could you expand on what you mean by that? I was thinking I would need to do something with Poisson statistics and think about the standard deviation with these small numbers etc. to try and figure if the source is variable, or if we should statistically expect such deviations.

Originally Posted by Guy

Warning: You should only be testing this if you were expecting to find a trend in time BEFORE you did the experiment. It is very bad form to test hypotheses suggested by the data on the same data set.

Well.. I originally looked at the data just to see if there were any kind of changes in output. I wasn't specifically looking for this downward trend. But I can't test my hypothesis on any other data because there are no other data. Isn't testing on this data the best I can do?

Thank you!
Jenny

I supposed, that going back to my original question... if I try and quantify how likely it is that my trend is a statistical fluctuation - isn't that a useful thing to be able to say in absence of another data set on which to test my hypothesis?

Originally Posted by jenny1988

Thank you for your response. When you say "fit a line to the data", could you expand on what you mean by that? I was thinking I would need to do something with Poisson statistics and think about the standard deviation with these small numbers etc. to try and figure if the source is variable, or if we should statistically expect such deviations.

I did a linear regression. It's the only thing I can think to do; the point obviously isn't that we think the decay is linear, but as far as detecting a downward trend it gets the job done.

Well.. I originally looked at the data just to see if there were any kind of changes in output. I wasn't specifically looking for this downward trend. But I can't test my hypothesis on any other data because there are no other data. Isn't testing on this data the best I can do?

You should know what statistics you are planning to do before you collect the data, generally speaking. If you thought there should be a trend either up or down, and the data isn't inspiring you to check for it, then it is probably okay.

(Red Part): Just because something is the best you can do doesn't mean that it is sound statistics. This is a big reason why scientists always wait to see if a result is replicable, regardless of how seemingly convincing a study might be.

It is very bad form to test hypotheses suggested by the data on the same data set.

Just because something is the best you can do doesn't mean that it is sound statistics.

ill throw in my 2 cents on this, although im not sure the discussion is really on topic for the original post.

The article, which is flagged as being inadequately sourced, claims that hypotheses suggested by the data are likely to be accepted. This feels spurious since classical statistics do not normally allow you to accept any hypothesis. if the statistic is in the acceptance region you simply fail to reject H0. Failure to reject is not evidence in favor of H0, and so there is no danger of "incorrectly accepting" anything. (CAVEAT:there are exceptions but im not aware of any that are relevent here)

In any event the reasoning would not seem to apply in this case as we would be choosing a hypothesis that we expect to be rejected (ie, that there is no trend).

As far as the original question goes, a linear regression sounds sensible as suggested by the previous poster. However there would be difficulties given the low volume of your data (13 data points may be too low for the CLT, for example). If i remember my OLS, You'd have to make a strong assumption about the distribution of your errors (eg, they are normal) which may distort your results, as you would effectively be testing the joint hypotheses that the errors are normal and there is no trend.

The poisson approach suggested by the OP could also be fun to explore, and would presumably give an exact test if you throw some algebra at it. However you would again be testing the joint hypothesis that the rate is constant and the underlying distribution is poisson.

A number of alternatives are suggested by this paper:
http://journals.tubitak.gov.tr/engin...4-5-0206-6.pdf

I didn't read it fully but it seems to suggest that you use either the Mann-kendell test or a t-test proposed by Haan (1977), and it doesn't really matter which. However you still need to be careful in using these method blindly because of your small dataset. I haven't checked if the statistics are only valid asymtotically, but you should if you plan to use them.

Originally Posted by SpringFan25

ill throw in my 2 cents on this, although im not sure the discussion is really on topic for the original post.

I wouldn't believe everything that's on wikipedia, especially articles which are flagged as needing verification. The article claims that hypotheses suggested by the data are likely to be accepted. This feels spurious since classical statistics do not normally allow you to accept any hypothesis. if the statistic is in the acceptance region you simply fail to reject H0. Failure to reject is not evidence in favor of H0, and so there is no danger of "incorrectly accepting" anything. (CAVEAT: i expect there are exceptions, eg in bayseian methods, but no one has proposed those methods here)

In any event the reasoning would not seem to apply in this case as we would be choosing a hypothesis that we expect to be rejected (ie, that there is no trend).

I don't think you understand which hypothesis you are risking accepting. You are risking accepting a false ALTERNATIVE, i.e. REJECTING A TRUE NULL by doing this. We look at this data and (1) plot the points, (2) notice a trend and so decide to (3) test that there is NO trend. This is essentially a mild form of data snooping, which I hope we all agree is bad.

The problems with this sort of analysis are well known, regardless of any problems you have with the wikipedia article. I didn't mention it because I read about it on wikipedia, I mentioned it because it is an issue with doing this kind of analysis and wikipedia happens to have an article on it.

(Red Part): Whether you want to say "accept" or "fail to reject" when talking about the null is really just a matter of terminology at heart. The advanced texts usually say "accept" (e.g. the canonical text by Lehman). Even in introductory classes, though, they tend to say things like "reject the null hypothesis" which is essentially the same as "accept the alternative."

You are risking accepting a false ALTERNATIVE, i.e. REJECTING A TRUE NULL by doing this.

Again, confining the discussion to classical statistics: the probability of rejecting a true null is given by the confidence level and is completely independent of how you selected the null&alternative.

Whether you want to say "accept" or "fail to reject" when talking about the null is really just a matter of terminology at heart.

I completely disagree. "Accept" would mean you believe the null model describes the data exactly. "Fail to reject" means that you did not find evidence against the null. Those statements are not logically equivalent except in very special circumstances which dont apply here.

I admit people say "accept" but they do so loosely and hopefully only as a shorthand for a more rigorous understanding. In the UK 1st year students routinely lose marks in exams if they say "accept" instead of "fail to reject". In my experience more advanced students are allowed to use the shorthand but only because they are assumed to understand that it cant be taken literally.


I do accept that if you think those two statements are equivalent then your previous argument is more plausble. But that is definately not how i was taught statistics. perhaps meanings differ in different countries?

Anyway, our discussion probably wont help the OP and i dont want to get an infraction, so i'll say nothing further. It might make in interesting new thread or PM if you are interested though

PS: my previous post was not intentionally edited after you replied. I hadn't seen yours as i was doing a long edit.

Originally Posted by SpringFan25

Again, confining the discussion to classical statistics: the probability of rejecting a true null is given by the confidence level and is completely independent of how you selected the null&alternative.

I'm not sure how to convince you otherwise, but this just isn't true. All of the classical setup is done under the assumption that you are setting confidence limits etc BEFORE looking at the data. Otherwise you would need to condition on the part of the data you looked at (in the independent case, this is tantamount to needing a new data set).

Why do you think people partition their data sets into "exploratory" and "confirmatory" sub data sets? Because they like loosing power to detect things?