Strictly speaking, the term "dimension" does not apply to matrices- it applies to vector spaces. However, a matrix is a linear transformation from one vector space to another. The "dimension" of the matrix can be thought of as the dimensions of those two vectors spaces- if a matric has, say, m rows and n columns, it is a linear transformation from a vector space, U, with dimension n to a vector space, V, of dimension m. Some texts will refer to "n by m" as the "dimension" of the matrix. On the other hand, we can think of the collection of all "n by m" matrices as a vector space in its own right- it has dimension mn.
If a matrix is "one-to-one", then it would map the vector space of dimension n to to a subspace of dimension n. In that case, the "rank" of the matrix is n. If it is not one-to-one, then it maps U, of dimension n, to a subspace of V of dimension less than n. (If m< n, then the matrix can't be one-to-one.) In either case, the rank of the matrix is the dimension of that subspace.