Yes, that is one direction. If is finite then there are only possible matrices (including those of determinant 0), as there are choices for each position in a given matrix.

To prove that if is an infinite field then is infinite, think about the matrix , where is the identity matrix and .