Technically you aren't looking for Newton's 2nd Law here, exactly, but another form of it known as the "Impulse-Momentum" theorem. The specific form you want is:
where is the average force on the hammer and is the time during which the hammer is decelerating.
where is the average acceleration.
I disagree with both the given answers here.
Part a) We know that the nail moves a distance of 0.45 cm when in contact with the hammer, so this is the distance the hammer moved from the time of contact to when the hammer stopped (assuming the hammer doesn't rebound, which it should.) We know the initial speed of the hammer, so assuming a constant deceleration, we can find the deceleration:
(For a coordinate system here I am taking as the origin the point where the hammer first strikes the nail and a +x direction in the direction that the hammer travels while "pounding" the nail.)
<-- Dropping the units for convenience
Taking the magnitude of this I get that
Part b) is done in exactly the same way and I get that
I note that the accelerations are 1000 m/s^2 and 3000 m/s^2, respectively, which are numerically the answers you listed. Perhaps someone copied the wrong numbers when giving you the answers?