Hint: Try drawing out it really helps, this is basically finding the missing side of a triangle, I believe
The smaller triangle has a base of x and a height of 5
The larger triangle has a base of 4+x and and unknown height, y.
Using the ratio, x/5 = (4+x)/y, solve for y and replace that into the formula
x^2 +y^2 = z^2
Take the derivative of z(x) = square root (x squared + y squared)
and find the critical points. Test for a min value and plug x back into
xsquared plus ysquared = zsquared to solve for z (lenght of ladder)
Sorry. That is super confusing without a diagram, but hopefully it somewhat helped.