I guess you are using it to determine the nature of an extrema?

Anyway, matrix is negative definite when you take its 1x1, 2x2, 3x3 ... nxn determinants and the sign goes - + - + - ...

You must take the determinant from the upper left.

If the "second order" (2x2) determinant is negative then the matrix is neither positive or negative definite. You have a saddle point.