First I'll recap your work (since I have to do it anyway), then I'll explain the issue:
Stationary points are where and , so where and .
The 2nd equation gives , so , so
The Hessian is
So , which is only 0 when x = 1/3. Thus both of those stationary points are non-degenerate.
Now the question, for each of the two stationary points (1, 1, -1) and (-1/3, -1/3, 5/27), is if H(f) positive definite (local min), negative definite(local max), or indefinite(saddle point)?
!!This is *not* entirely knowable by looking at the sign of the determinant of the Hessian. Ex: has determinant 1, but is negative definite!
Local max/min/saddle for f at a critical point are about the eigenvalues of the Hessian, not its deternimant. All positive means local min, all negative means local max, and mixed (some positive, some negative) means saddle point. None are zero, since "non-degenerate stationary point iff Hessian has non-zero determinant iff it has no zero eigenvalues."
In the special case of a 2x2 matrix, since there are only two eigenvalues (counting multiplicity), and neither of them is zero, and the determinant is their product, if the determinant is positive, then either they're both positive or both negative. If the determinant is negative, then one is positive and the other is negative.
Thus *only* in the special case of a 2x2 matrix, you can say this:
If f has a nondegenerate stationary point p, and Det(H(f)) < 0 at p, then p is a saddle point of f, and if Det(H(f)) > 0 at p, then p is either a local maximum or local minimum.
Remember, that's *only* in the special case of a 2x2 matrix. In higher dimensions, the first part (Det(H(f)) < 0 implies saddle point) generalizes to even dimensions, and the second part (Det(H(f)) > 0 implies it's either a local max or a local min) simply fails in general.
(Note: real symmetric matricies always have all real eigenvalues (counting mutliplicity)).
OK, so for this problem, need to examine the eignvalue for H(f) at (1, 1, -1) and (-1/3, -1/3, 5/27).
Get characteristic polynomial first, then eigenvalues:
Thus the eigenvalues of H(f) at (-1/3, -1/3, 5/27) are .
Therefore f has a saddle point at (-1/3, -1/3, 5/27).
Now for (1, 1, -1): . You get:
Thus the eigenvalues of H(f) at (1, 1, -1) are , which are both positive, making H(f) positive definite there.
Therefore f has a local minimum at (1, 1, -1).