O(h) essentially means that the quantity goes to zero at the same rate as h. Or that the quantity is similar in magnitude as h.

The x<x^2 is coming from the 2x in f(x). Here they are saying that when x>1 we know that x^2>x and therefore 2x^2 > 2x. You can then apply this reasoning to the constant in f(x), i.e. when x>1 x^2>1. Putting all this together we have that x^2+2x+1 < x^2 +2x^2 +x^2 = 4x^2.