has non-trivial solution then either a=6 or b=-2
find the general solution to the system of equations when a=6
In your classnotes or textbook it will say somewhere that for a homogeneous system of linear equations to have a non-trivial solution the determinant of the coefficient matrix has to equal zero. (In which case there is an infinite number of solutions).