Hmm, gradient descent should find local minima by definition. That's one of it's pitfalls, kinda, that it gets caught in local minima instead of finding the global.
Did you try using a smaller step size? What kind of functions are you using?
I've been working on a Java program to do gradient descent on 4 different functions.
This is what I'm doing:
- Calculating a step size ( I need to do an adaptive step size, but for now I'm just using a small constant like 0.1).
- Calculating the gradient of the function numerically
- Multiplying the gradient by the step size
- Adding that product to the current point. (i.e. p + where is step size, p is current point, and is numerical gradient of the function.
- Repeating until the distance between the points is less than a certain threshold (like 10e-6).
Now my question is - how do I search for the local minimum and not the global minimum?
In other words, some of my initial guesses just lead my algorithm to go to negative infinity. How can I tell if it's going to infinity or if going to a local minimum?