If we have to do three digit rounding of
0.266666666667
then as per my understand answer should be
0.267
but book says 0.266
Why?
Rounding to 3 digits is 271 (round down because the next digit is a 4) whereas rounding to 3 decimal places is 271.427 (round up because the next digit is 8)
Which do you mean?
Whether you have it as $\displaystyle 271.427 \text{ or } 0.271427 \cdot 10^3$ is up to you since they're both equivalent.
What exactly is the statement of the problem? If the problem said "round to three decimal places" then what you have is correct. But if it just said "round to three digits", then the correct answer is just "271". Those are also "three digits".