Here's (a) in two parts. Had to convert to raw format to paste it here. I think you can follow it right? The D[f[x,y,z],x] is just the partial with respect to x. Same dif for others. The syntax:

extremaInUnitBall = Select[myExtrema,

Sqrt[#1[[1]]^2 + #1[[2]]^2 +

#1[[3]]^2] < 1 & ]

means "select from the table of myExtrema, those values for which $\displaystyle \sqrt{x^2+y^2+z^2}<1$

Code:

f[x_, y_, z_] := Sqrt[48]*x*y*z -
(1 - x^2 - y^2 - z^2)^(3/2)
g[x_, y_, z_] := x^2 + y^2 + z^2 - 1
myExtrema = {x, y, z} /.
Solve[{D[f[x, y, z], x] == 0,
D[f[x, y, z], y] == 0,
D[f[x, y, z], z] == 0}, {x, y, z}]
extremaInUnitBall = Select[myExtrema,
Sqrt[#1[[1]]^2 + #1[[2]]^2 +
#1[[3]]^2] < 1 & ]
{{-1, 0, 0}, {0, -1, 0}, {0, 0, -1},
{0, 0, 0}, {0, 0, 1}, {0, 1, 0},
{1, 0, 0}, {-(Sqrt[3]/5), -(Sqrt[3]/5),
-(Sqrt[3]/5)}, {-(Sqrt[3]/5),
Sqrt[3]/5, Sqrt[3]/5},
{Sqrt[3]/5, -(Sqrt[3]/5), Sqrt[3]/5},
{Sqrt[3]/5, Sqrt[3]/5, -(Sqrt[3]/5)}}
{{0, 0, 0}, {-(Sqrt[3]/5), -(Sqrt[3]/5),
-(Sqrt[3]/5)}, {-(Sqrt[3]/5),
Sqrt[3]/5, Sqrt[3]/5},
{Sqrt[3]/5, -(Sqrt[3]/5), Sqrt[3]/5},
{Sqrt[3]/5, Sqrt[3]/5, -(Sqrt[3]/5)}}

Second part to (a) using Lagrange multipliers. Again, the variable myLagrange is just the table of values for which the four Lagrange equations are zero. Then the syntax

(f[#1[[1]], #1[[2]], #1[[3]]] & ) /@

myLagrange

means, use the "pure function" (f[#1[[1]], #1[[2]], #1[[3]]]&) and go throgh the table myLagrange and substitute the first three elements in each term into the function f(x,y,z). From that list I get the max is 4/3 and the min is -4/3.

Code:

In[83]:=
myF[x_, y_, z_, \[Lambda]_] := f[x, y, z] +
\[Lambda]*g[x, y, z];
myLagrange = {x, y, z, \[Lambda]} /.
Solve[{D[myF[x, y, z, \[Lambda]], x] == 0,
D[myF[x, y, z, \[Lambda]], y] == 0,
D[myF[x, y, z, \[Lambda]], z] == 0,
D[myF[x, y, z, \[Lambda]], \[Lambda]] == 0},
{x, y, z, \[Lambda]}]
(f[#1[[1]], #1[[2]], #1[[3]]] & ) /@
myLagrange
Out[84]=
{{-(1/Sqrt[3]), -(1/Sqrt[3]), 1/Sqrt[3],
-2}, {-(1/Sqrt[3]), 1/Sqrt[3],
-(1/Sqrt[3]), -2}, {1/Sqrt[3],
-(1/Sqrt[3]), -(1/Sqrt[3]), -2},
{1/Sqrt[3], 1/Sqrt[3], 1/Sqrt[3], -2},
{-1, 0, 0, 0}, {0, -1, 0, 0},
{0, 0, -1, 0}, {0, 0, 1, 0},
{0, 1, 0, 0}, {1, 0, 0, 0},
{-(1/Sqrt[3]), -(1/Sqrt[3]),
-(1/Sqrt[3]), 2}, {-(1/Sqrt[3]),
1/Sqrt[3], 1/Sqrt[3], 2},
{1/Sqrt[3], -(1/Sqrt[3]), 1/Sqrt[3],
2}, {1/Sqrt[3], 1/Sqrt[3],
-(1/Sqrt[3]), 2}}
Out[85]=
{4/3, 4/3, 4/3, 4/3, 0, 0, 0, 0, 0, 0,
-(4/3), -(4/3), -(4/3), -(4/3)}