I'm trying to solve such a problem that says:
A= [a,b;0,1] "2x2 matrix"
I've tried A,A^2,A^3 to get the formula but I couldn't
Either way, as you can see here, your equality with the 3rd part does not hold:
WolframAlpha - counter example