"Reflection in a line" maps any vectorparallelto the line into itself. It maps any vector perpendicular to the line into its negative.

So: What is a vector in the direction of the line ax+ by? What is a vector in the direction of ax+ by= 0? What is a vector perpendicular to it?

Another way: find the rotation matrix, A, that rotates ax+ by= 0 to y= 0. The matrix that refleclects about y= 0 is . The matrix that reflects in the line ax+ by= 0 is .