First, an inner product can be negative. (No inner product of a vector with itself can be negative.) In fact the standard dot product is an inner product.
This problem is an exercise in using subscripts. Compare the standard inner product on the nxn matrices with the definition of trace([B^t]A).