The t-test itself, tests the sample mean, which will be normally distributed.

If your data itself is skewed but n is large enough the t-test is considered a robust test.

I would recommend using a t-test and then using a different test, maybe a non-parametric test like the Kolmogorov-Smirnov test.

Are the data sets considered different in both tests?