You don't pick a sample size and repeatedly find the mean with many sets of data with that sample size. You put all of your samples into one data set and the size of that is the sample size that affects the accuracy of the sample mean. Does this clear things up?