Have you tried using a truncated mean?
1. Sort the entries in your matrix using the function sort.
2. Remove the first and last entries of the matrix.
3. Calculate the mean of what you have left.
If the number of outliers doesn't vary too much, you should be able to find values for and that get rid of most of the unwanted values.
Edit: There is a MATLAB function for calculating the truncated mean, called trimmean.