Results 1 to 4 of 4

Math Help - Basic steps of NLLS regression

  1. #1
    Member
    Joined
    Nov 2008
    Posts
    92

    Basic steps of NLLS regression

    Hi,

    I am having trouble implementing the most basic NLLS algorithm. I am following the example from here to every detail, yet my implementation keeps on diverging after 5 or so iterations. I will give an outline of what I'm doing below.

    The example tries to fit the data to a Gaussian function, so the paramters for my model are params = [A, x_{o}, \sigma].

    Step 1
    Make an initial guess of the paramters

    A^0 = 0.8

    x_{o}^0 = 15

    \sigma^0 = 4


    Step 2
    Compute the values of the Jacobian using the current paramter values for each data point i=1,2,...,m

    J_{i,A} = \exp^{-((x - x_{0}^2)/2\sigma^2)}

    J_{i,x_{0}} = A(x - x_{0})\exp^{-((x - x_{0}^2)/2\sigma^2)}/\sigma^2

    J_{i,\sigma} = A(x - x_{0})\exp^{-((x - x_{0}^2)/2\sigma^2)}/\sigma^3


    Step 3
    Compute the value of the current model approximation

    \hat{f} = A\exp^{-((x - x_{0}^2)/2\sigma^2)}

    Error between the data set and the current model

    \Delta\beta = y - \hat{f}

    If the error is large, update the paramter values as follows

    \Delta{params} = (J^T J)^{-1} (J^T \Delta\beta)

    params^{k+1} = params^k + \Delta{params}

    Loop back to Step 2.

    If my pseudo-code has confused things, please ignore it and just describe how such a basic NLLS algorithm should be implemented.

    Thanks!
    Last edited by algorithm; October 11th 2012 at 12:53 PM.
    Follow Math Help Forum on Facebook and Google+

  2. #2
    MHF Contributor
    Joined
    Sep 2012
    From
    Australia
    Posts
    3,607
    Thanks
    591

    Re: Basic steps of NLLS regression

    Hey algorithm.

    I don't have deep working knowledge of these kinds of solution finders, but you might want to look at something that has been formalized and looked at over the past couple of decades, the EM algorithm is a good one to look into:

    Expectation

    In terms of your actual issues, this is going to (if your code has been correct) be an issue of looking into the stability of your various scheme.

    In terms of Normal distribution fitting, the EM has a lot of implementations to do this including in packages like R and if you can look at the R code for this algorithm, then you can get a much better idea of what's going on and compare it to your own implementation:

    The R Project for Statistical Computing

    Now a search on the implementation brought up a thread like this and after reading this I am hesitant to offer specific advice, so I will just post the thread which should give you food for thought:

    R help - EM algorithm

    For a normal model (as in normal PDF), there was a paper that implemented EM in R that uses bootstrapping in combination with the EM:

    http://homepage.stat.uiowa.edu/~kcow...7/chang166.pdf
    Follow Math Help Forum on Facebook and Google+

  3. #3
    Member
    Joined
    Nov 2008
    Posts
    92

    Re: Basic steps of NLLS regression

    Hi chiro,

    Thanks for your input and the links.

    What I must use specifically is NLLS, and what's more I need to generalise it to non-Gaussian functions. The reason I am testing using a Gaussian function is because I know what the solution should look like if the algorithm works.
    Follow Math Help Forum on Facebook and Google+

  4. #4
    MHF Contributor
    Joined
    Sep 2012
    From
    Australia
    Posts
    3,607
    Thanks
    591

    Re: Basic steps of NLLS regression

    Did you run a search on non-linear optimization code examples (specific implementations) that do this?

    There is bound to be one out there for this purpose (and many statistical packages do this specifically for a variety of distributions).

    Google turned up this:

    Levenberg-Marquardt in C/C++
    Follow Math Help Forum on Facebook and Google+

Similar Math Help Forum Discussions

  1. Which steps to use and when to use them ? ?
    Posted in the Algebra Forum
    Replies: 1
    Last Post: September 18th 2011, 09:59 PM
  2. Ncaaf multiple regression/ stepwise regression
    Posted in the Advanced Statistics Forum
    Replies: 0
    Last Post: August 10th 2011, 06:21 AM
  3. Basic Matlab help. Can't seem to grasp basic concept
    Posted in the Math Software Forum
    Replies: 5
    Last Post: February 28th 2010, 08:12 PM
  4. Replies: 6
    Last Post: November 6th 2009, 09:28 PM
  5. Regression SS in multiple linear regression
    Posted in the Advanced Statistics Forum
    Replies: 0
    Last Post: June 20th 2009, 12:23 PM

Search Tags


/mathhelpforum @mathhelpforum