You are trying to minimize the function (the regular Euclidean norm on ) subject to the constraint . However, minimizing is the same as minimizing . (Do you see why?)

I would use Lagrange multipliers:

So you need to solve the system:

This system is not nearly as hard as it looks. Note that if , then as well (from the first three equations), but this contradicts the fourth equation, because we'd get . So . But from the second and third equations, this implies that . Now our system is simpler:

Solving this is easy. We get (which is irrelevant for this problem) and .

So the points closest to the origin are and , just like the answer key says.

If you think about this geometrically, it makes sense. The points closest to the origin will be on the minor axis of the ellipsoid, which in this case is the x-axis, and can be obtained by plugging in for and solving for .