Do it one step at a time. Any linear transformation on can be represented as a 2 by 2 matrix. And you want
Do that multiplication on the left, set components equal to those on the right and you can see what a and b must b.
You could do the same thing for the rotation but it is simpler to use the fact that a counter clockwise rotation, about the origin, through is represented by the matrix
Doing both is equivalent to multipying the matrices. Be sure you multiply in the right order: the matrix applied first on the right.