The momentum of the hammer when it reaches the nail is

mv = Ft = mg.(2h/g)^1/2 = m(2gh)^1/2

using x = vi*t + 0.5gt^2 (x-displacement, vi-initial velocity=0)

The hammer does not rebound after the collision, i.e. it's final velocity is zero, so using conservation of momentum, the nail's momentum after the collision must be the same as the hammer's just before the collision:

Mv = m(2gh)^1/2

So it's velocity will be: (m/M)(2gh)^1/2

The nail is moving against a constant resistive force F, i.e. this force causes a negative acceleration of F/M on the nail. Using the formula

vf^2 = vi^2 + 2ax (vf-final velocity=0, a-acceleraion=-F/M)

Then re-arrange for displacement x:

x = (vf^2 - vi^2)/(2a) = [0-((m/M)(2gh)^1/2)^2]/(2*(-F/M)) = (m/F)(gh/2)^1/2

Note that this makes sense: the distance the nail is driven in increases if you make m (hammer's mass), g (gravitational acceleration) or h (height hammer falls through) larger, but decreases if the resistive force F is increased.

Not sure I understand this part correctly. Are they saying: show that raising the hammer through a vertical distance h requires energy independent of h? That's obviously nonsense.Show that the total energy expended in raising the hammer during the operation of driving the nail fully into a depth of d is independent of the value of h and can be decreased by making the hammer more massive.

I can't figure out what this part of the question is asking-- can you give clarification please?