1 Attachment(s)

Fourier Transform Question: Fitting Data to a Sine-like Curve

I have some data (18 points) which are plotted on the attached image, f(x), the red line.

I do *not* know if this line can be described by a combination of known functions, but I am trying to find out.

It looks somewhat similar to sin(x), the blue line, and tan(x), the green line.

At this point I am trying random combinations of functions to see if I can find a function that accurately describes f(x):

1) f(x) = M*x + B + C*sin(x) + D*cos(x)

2) f(x) = M*x + B + sin(C*x) +cos(D*x)

3) f(x) = M*x + B + C*sin(x)*cos(x)

4) f(x) = M*x + B + C*sin^{2}(x)*cos(x)

5) f(x) = M*x + B + C*sin(x)*cos^{2}(x)

etc., etc.

where M, B, C, and D are real constants.

I am pretty much just making random guesses hoping to get lucky.

However, since the function looks so close to a portion of a sine curve, I am wondering if a Fourier Transform might reveal something. Assuming f(x) is composed of a mix of sine and cosine terms, isn't the whole point of a Fourier Transform to reveal these separate components?

This is new territory for me; I was briefly exposed to Fourier Transforms during my undergraduate engineering studies but, since then, haven't touched them.

So . . . I have lots of questions.

If I perform a Fourier Transform on the data, even if it does not reveal the complete function, will it at least reveal if f(x) includes a linear term (i.e. - M*x + B)?

Does a Fourier Transform reveal only *linear* terms of cosine and sine? In other words, if the actual function were to take a form such as equations 4 and 5 above (which includes a power of 2), would a Fourier Transform reveal it?

Do I need more than 18 data points to make this work? If so, I will have to go back and get some more data points.

Any suggestions or advice is appreciated.

Attachment 29732

Re: Fourier Transform Question: Fitting Data to a Sine-like Curve

Hey DavidB.

If you want to generate a fourier series you can "project" a data set (or signal) to an orthogonal basis which is made up of sines and cosines (as well as the linear term corresponding to 1 or cos(0)).

The assumption is though that your function is periodic in that it satisfies f(x+a) = f(x) for some value of a that is finite.

If you want to do this then you need to calculate the appropriate integral using your data points where the projection function involves the integral of d*sin(ax)*f(x)dx [e,f] where the constants all depend on the location of your function, the wavelength and the harmonic.

Re: Fourier Transform Question: Fitting Data to a Sine-like Curve

Hi !

I doudt that FT would simplify the gessing process to find a convenient function.

It should be easier to give a more supported coment if the data was given on numerical form instead of graphical.

Re: Fourier Transform Question: Fitting Data to a Sine-like Curve

If you think its purely sinusoidal (or a combination of sinusoids) then the fourier series representation should do exactly that.

Re: Fourier Transform Question: Fitting Data to a Sine-like Curve

Quote:

Originally Posted by

**chiro** If you think its purely sinusoidal (or a combination of sinusoids) then the fourier series representation should do exactly that.

I agree with you if it is assumed that the function is purely periodic.

But from the wording of the question raised by DavidB, the function is likely to include non-periodic terms. That is why I think that some preliminary tests might be carried out before giving any advice. This cannot be correctly done only based on a graph.

Moreover, if there are 18 points distributed on only a fraction of period, such a data appears rather small for FT. Different methods might be more convenient.

Re: Fourier Transform Question: Fitting Data to a Sine-like Curve

What kind of analyses did you have in mind?

Re: Fourier Transform Question: Fitting Data to a Sine-like Curve

Quote:

Originally Posted by

**chiro** What kind of analyses did you have in mind?

Check if DavidB correctly did the regressions with the guessed functions.

Try simpler functions (not periodic) and see if better fit is obtained than with periodic functions.

Try non-linear regressions with some functions using the method from the paper :

Régressions et équations intégrales

All this can be done very quickly with available software.

2 Attachment(s)

Re: Fourier Transform Question: Fitting Data to a Sine-like Curve

Quote:

Originally Posted by

**JJacquelin** Hi !

. . .

It should be easier to give a more supported coment if the data was given on numerical form instead of graphical.

Hi, "JJacquelin".

I have attached two text files containing the data with which I am working: one contains the x-coordinates, the other contains the corresponding y-coordinates.

Note that I am working over the range 0 - 1 only.

(Also note that I added several more data points: instead of only 18 points, there are now 32.)

Quote:

Originally Posted by

**chiro** If you think its purely sinusoidal (or a combination of sinusoids) then the fourier series representation should do exactly that.

Hi, "chiro".

I don't know for sure if it is purely sinusoidal. And the range I am looking at is only from 0 - 1. But based on the curve, which is tantalizingly close to a sine wave over this range, I am *hoping* it might resolve to an equation that can be stated in terms of sines and cosines.

At the moment, I do not have access to commercial software that does Fourier series. Can you suggest a free online utility that can do this? (Seems like there are many powerful tools online; I will do a search myself, but if anybody in these forums can recommend a good one, from personal experience, I would like to know about it.)

Thanks all.

1 Attachment(s)

Re: Fourier Transform Question: Fitting Data to a Sine-like Curve

Hi DavidB

Just looking at the points seems like you have an exponential function, that you could express, for example, in the form:

$\displaystyle f(x;\tau) = 1-\exp{(-x/\tau)}$

where $\displaystyle \tau$ is a constant (in units of $\displaystyle x$) that tells at what $\displaystyle x$ you would expect to have a change in the $\displaystyle y$ magnitude of ~0.63 $\displaystyle (1-1/e) $. For your data, $\displaystyle \tau $ has a value of 0.88

Fitting this data is something easy. Generally, most software platforms contain some functions to fit to your data. They use the Levenberg-Marquardt algorithm (LM) to fit the data to the model. If the function of interests is not included in your software , you could try to fit with LM by defining a custom routine in your favourite programming language.

I uploaded an example of such implementation in Python that you can see here. Bellow you can find a plot with the results of the fitting.

Attachment 29752

PS: I would be curious to see how to fit this with a Fourier transform, I have never used it for that.

2 Attachment(s)

Re: Fourier Transform Question: Fitting Data to a Sine-like Curve

Thanks, Jose.

That curve does, indeed, look very close to the curve I have.

Attached are two more plots: one is the original curve, and the regression, on the same plot. Original y in blue. Regression in red.

They are extremely similar.

The other image is a plot of the original curve minus the regression.

(I suppose it could be flipped: plot the regression minus the original curve. In this case, it would look like a skewed sine wave: goes from 0 to a maximum, then crosses 0, goes negative, then back up to 0. Again, it seems to hint at a sine-like component).

I am going to try a few more things; hopefully, something will fall into place.

Attachment 29758

Attachment 29759

1 Attachment(s)

Re: Fourier Transform Question: Fitting Data to a Sine-like Curve

Hi DavidB !

Sorry for my late answer. I have been away for a few days.

There is no need for complicated functions. A very good fitting is obtained with polynomials of rather low degrees (Figure in attachment).

The fitting with some simple functions involving exponential for example can be rather good, but less than with the polynomials.

1 Attachment(s)

Re: Fourier Transform Question: Fitting Data to a Sine-like Curve

If you prefer sinusoidal functions (Figure in attachment), the fittig can also be good, but less than with polynomial. For example with y = a+b*sin(w*x)+c*cos(w*x) the Mean Square Deviation = 0.00130 is about 5 times worse than with the 4th degre polynomial MSD= 0.00028 and almost the same than the 2nd degre polynomial MSD= 0.00125

Comments about fitting to sinusoidal functions:

It was observed during the computation that the determinant of a matrix is very close to zero. This makes think that the function considered is not a sum of undependant terms. This is confirmed , on the attached figure, by observing that the values (c) is very close to (-a). So, the function might be y=a*(1-cos(w*x))+b*sin(w*x) which involves only 3 parameres to be optimized instead of 4. Nevertheless, the fitting (MSD= 0.00133) is not so good than with polynomials.

Re: Fourier Transform Question: Fitting Data to a Sine-like Curve

Hi JJacquelin!

These are nice fittings! If I'm not restricted to any model, I tend to fit data to the easiest function (i.e the one with least number of parameters). The last thing i tend to use is a polynomial fit, because, to my experience, every data can be fitted to a polynomial :)

I guess the question is... what does DavidB have in mind to fit the function, and what is the nature of the data.

1 Attachment(s)

Re: Fourier Transform Question: Fitting Data to a Sine-like Curve

Hi jguzman !

A lot of fonctions can fit a curve of this kind. For exemple the Weibull function (attachment) and many other. I don't intend to advice to chose a particular one among them. I only report rough results and some general tendencies. It's up to DividB to chose, depending on his criteria : the precision of fitting, or the simplicity of the computations, or the preference for a category of functions, or something else. Only DavidB knows about the context of the problem.

2 Attachment(s)

Re: Fourier Transform Question: Fitting Data to a Sine-like Curve

Quote:

Originally Posted by

**JJacquelin** . . .

It was observed during the computation that the determinant of a matrix is very close to zero. This makes think that the function considered is not a sum of undependant terms. This is confirmed , on the attached figure, by observing that the values (c) is very close to (-a). So, the function might be y=a*(1-cos(w*x))+b*sin(w*x) . . .

Hi, " JJacquelin".

Thank-you very much.

(By the way, what is your first name, so I do not have to address you by your forum handle?)

I am curious: what software did you use to compute the regressions and create those plots? It looks like it includes some powerful features.

It is intriguing that the sine regression indicates a factor of [1 - k cos(x)]; I was actually thinking the function might include this factor--but in the denominator. Putting a factor of 1/[1 - k cos(x)] seems to "pull" a function to one side. For example, if you plot sin(x)/[1 - k cos(x)], where 0 < k < 1, the resulting function differs from the regular function by having its maxima and minima moved over somewhat.

In addition to considering polynomial regressions of power 2 and 4, I also did a linear regression (linear least-squares data fit). The result was a function of the form y = 0.6948875476794*x + 0.0505798342033.

I also tried something even less sophisticated; instead of doing a linear least-squares fit, I drew a straight line between the endpoints. The result is a function of the form y = 0.67361202918322*x.

This straight line, and the original y-curve, is plotted on the attached graph, y_and_line1. Original data, y, is red for comparison.

The difference between these two curves is plotted on the second attached graph, y_minus_line1. The second graph looks very close to being symmetric around the mid-point (x = 0.5), but it doesn't quite work out. A couple quick calculations with a hand calculator equal distances from x = 0.5 indicate slightly different y-values. Otherwise, I had hoped to find a second degree curve.

These data points that make the y curve are computed theoretically; they are not empirical numbers gathered from an experiment, so I am hoping to find an exact solution.

Any and all additional suggestions for functions would be welcome; if anything else comes to mind, please post back.

Thanks for everybody's input. I have a lot to work with now, so will take some time to mull over all this information.

Attachment 29795Attachment 29796