If we have to do three digit rounding of
0.266666666667
then as per my understand answer should be
0.267
but book says 0.266
Why?
What exactly is the statement of the problem? If the problem said "round to three decimal places" then what you have is correct. But if it just said "round to three digits", then the correct answer is just "271". Those are also "three digits".