# Thread: Find point of deviation from reference

1. ## Find point of deviation from reference

Hi,
I have a set of reference values. After a particular experiment I get a
set of result.
How can I identify a point after which it deviated above a threshold
value?

An example: suppose the reference is 20,19,18,17,16,15,14,13,12,11.
After an experiment the values are changed to :
20,19,18,17,16,11,10,9,8,7.
At the 6th point the value deviated from reference by a value of 5.
After what operations can I get the answer as "6"?

2. I'm not sure I understand what you're about. At the 6th point, the value has changed by 4, not 5: 15 - 11 = 4.

After what operations can I get the answer as "6"?
What are the sorts of "operations" that you're allowed to have? Are these data points results from an experiment? If so, you're not going to change the values except by changing the experiment.

At the 6th point, the value has changed by 4. Not 5. A typo error;

The results are the outcome of an experiment. we can have any sort of operations but finally have to get the point from it deviated. the values may not be the same but the experiment remains the same.

4. I still have no clue what you're trying to do. Let me illustrate by quoting you:

we can have any sort of operations
Ok. Heart transplant with a triple-bypass, mathematical pointwise addition or multiplication, driving a car, pointwise exponentiation, swapping hard drives, list concatenation, ...

My point is that there has to be a universe of discourse of operations from which you can pick ones that are valid (obviously, some of the ones I listed above are not valid on a list of numbers, while others are). What is your universe of discourse here? That is, please list, concretely, all the possible operations which you are considering using on these lists to manipulate the data.

but finally have to get the point from it deviated.
This is not correct English: it doesn't parse. Can you please rephrase using complete sentences? If you can also illustrate with a picture, that'd also help.

What I meant by "Any sort of operations" is that we can perform mathematical operations(eg. differentiation) to the data to get the intended results.
I am attaching 2 images, one is the reference and other after an expt.

6. Originally Posted by mailranjithr
What I meant by "Any sort of operations" is that we can perform mathematical operations(eg. differentiation) to the data to get the intended results.
(Drily) Not exactly the scientific method to alter data after obtaining it, except in a well-documented fashion and for a specific purpose. Also, you can't technically ever differentiate measured data, because all measured data is necessarily discrete. You can maybe try some sort of discrete derivative, though, like a difference.

I am attaching 2 images, one is the reference and other after an expt.
I went through both images, and found that my understanding did not increase at all.

7. In that expt.jpg we can see a sharp decline by a value > 4 after the point 32 in X-axis. This is not seen in the ref.jpg. Both ref and expt values are sampled and we see the data in CSV format. So using the data and doing some operations, say in MATLAB, we should get that point as a peak. You can suggest me a way to get that peak...
Thanks

8. Now we're getting somewhere. You want to automatically detect the location of a steep drop in the signal. Is that correct? What assumptions can you make about the signal on either side of this drop? In particular, can you assume that the data is pretty much a straight line on either side of the drop?

9. Yes. I want to automatically detect the location of a steep drop in the signal....
The data on both side of the drop is much like a straight line like in the image expt.jpg.

10. Ok. One option is the following:

1. Take the discrete differential of the signal. That is, from the list of data points $\displaystyle \{x_{k}\},$ where $\displaystyle 0\le k\le n,$ construct the new list

$\displaystyle \Delta x_{j}=x_{j}-x_{j-1},$ for $\displaystyle 1\le j\le n.$ Notice you will necessarily lose one point in this process, because of the fence-post principle.

2. The drop you're looking for will then appear as a negative peak in the differential signal. If your peak-finding algorithm prefers positive peaks, then simply multiply your differential signal by -1.

And that's it. Your peak-finding algorithm might be as simple as finding the maximum point, if you're sure that there will only be one such large drop.

On the other hand, the operation of taking the differential is highly sensitive to noise. (Integration smooths functions out, and is not sensitive to noise; differentiation makes functions more angular, and is sensitive to noise). So if you tend to get a lot of noise in the signal, you might get a large drop that isn't "real" - it isn't the drop you're looking for. One way to compensate for this is to run your data through some sort of running average filter before taking the differential. Positive: makes your algorithm more robust to noise. Negative: this operation will tend to spread your sudden drop so that it "takes longer". That is, your final result is not likely to be as accurate as it might be.

These are trade-offs you'll have to consider. Hope this helps.