I have no idea what you mean by " changed the system into fractional part". What did you get when you did that?
Start looking for a "general proof" by looking at special cases. Obviously "0" and "0^2" so [0^2]- [0]^2= 0 and those two functions will be the same (so the total function will continue to be 0 and, of course, be continuous) until one or the other of them changes value. When does that happen? Well, they stay equal to 0 until either x or x^2" is equal to 1- which happens "at the same time"- when x= 1. So the total will still be 0 (and, of course, be continuous) until x= 2 or x^2= 2. That happens when x= \sqrt{2}. At that point [x^2]= 2 while [x]^2= 1, still. At x= \sqrt{2} the function value changes from 0 to 1- 0= 1 and so the function is discontinuous there. But it retains that value until x= \sqrt{3} when [\lfloor x^2\rfloor= 3 and the function value becomes 3- 1= 2. Do you see what is happening and what the points of discontinuity are?