With this question do I simply replace x in the integral, simplify and integrate?
"Using the substitution x=(a+b)/2 +((b-a)/2)cos t, evaluate Integral from a to b of square root of((x-a)(b-x))dx".
Do I need new limits?
You have $\displaystyle \displaystyle \begin{align*}\int_a^b{\sqrt{(x-a)(b-x)}\,dx} &= \int_a^b{\sqrt{-x^2 + (a+b)x - ab}\,dx} \\ &= \int_a^b{\sqrt{-\left[x^2 - (a+b)x + ab\right]}\,dx} \\ &= \int_a^b{\sqrt{-\left\{x^2 - (a + b)x + \left[-\left(\frac{a+b}{2}\right)\right]^2 - \left[-\left(\frac{a+b}{2}\right)\right]^2 + ab\right\}}\,dx} \\ &= \int_a^b{\sqrt{-\left[\left(x - \frac{a+b}{2}\right)^2 - \frac{a^2 - 2ab + b^2}{4}\right]}\,dx} \\ &= \int_a^b{\sqrt{\left(\frac{a - b}{2}\right)^2 - \left(x - \frac{a+b}{2}\right)^2}\,dx}\end{align*}$
So now make the substitution $\displaystyle \displaystyle x - \frac{a+b}{2} = \left(\frac{a-b}{2}\right)\cos{\theta} \implies dx = -\left(\frac{a-b}{2}\right)\sin{\theta}\,d\theta $,
also noting that when $\displaystyle \displaystyle x = a$
$\displaystyle \begin{align*}a - \frac{a+b}{2} &= \left(\frac{a-b}{2}\right)\cos{\theta} \\ \frac{a-b}{2} &= \left(\frac{a-b}{2}\right)\cos{\theta} \\ 1 &= \cos{\theta} \\ \theta &= 0\end{align*}$
and when $\displaystyle \displaystyle x = b$
$\displaystyle \displaystyle \begin{align*} b - \frac{a+b}{2} &= \left(\frac{a-b}{2}\right)\cos{\theta} \\ -\left(\frac{a-b}{2}\right) &= \left(\frac{a-b}{2}\right)\cos{\theta} \\ -1 &= \cos{\theta} \\ \theta &= \pi \end{align*}$
So the integral becomes
$\displaystyle \displaystyle \begin{align*} \int_a^b{\sqrt{\left(\frac{a - b}{2}\right)^2 - \left(x - \frac{a+b}{2}\right)^2}\,dx} &= \int_0^{\pi}{\sqrt{\left(\frac{a - b}{2}\right)^2 - \left[\left(\frac{a-b}{2}\right)\cos{\theta}\right]^2}\left[-\left(\frac{a-b}{2}\right)\sin{\theta}\right]\,d\theta} \\ &= -\left(\frac{a-b}{2}\right)\int_0^{\pi}{\sin{\theta}\, \sqrt{ \left( \frac{a-b}{2} \right)^2 - \left( \frac{a-b}{2} \right)^2 \cos^2{\theta} } \, d\theta} \\ &= -\left(\frac{a-b}{2}\right) \int_0^{\pi}{\sin{\theta}\, \sqrt{ \left( \frac{a-b}{2} \right)^2\left( 1 - \cos^2{\theta} \right) } \,d\theta } \\ &= -\left( \frac{a-b}{2} \right) \int_0^{\pi}{ \sin{\theta} \, \sqrt{ \left( \frac{a-b}{2} \right)^2 \sin^2{\theta} } \,d\theta} \\ &= -\left(\frac{a-b}{2}\right) \int_0^{\pi}{\sin{\theta} \left( \frac{a-b}{2} \right) \sin{\theta}\,d\theta} \\ &= -\left(\frac{a-b}{2}\right)^2 \int_0^{\pi}{ \sin^2{\theta} \,d\theta }\end{align*}$
$\displaystyle \begin{align*} &= -\left(\frac{a-b}{2}\right)^2 \int_0^{\pi}{ \frac{1}{2} - \frac{1}{2}\cos{2\theta} \,d\theta} \\ &= -\left(\frac{a-b}{2}\right)^2 \left[\frac{1}{2}\theta + \frac{1}{4}\sin{2\theta} \right]_0^{\pi} \\ &= -\left(\frac{a-b}{2}\right)^2 \left[\left(\frac{\pi}{2} + \frac{1}{4}\sin{2\pi}\right) - \left(\frac{0}{2} + \frac{1}{4}\sin{0}\right)\right] \\ &= -\frac{\pi}{2}\left(\frac{a-b}{2}\right)^2 \\ &= -\frac{\pi (a-b)^2}{8}\end{align*}$
Geometric interpretation: the integrand function is $\displaystyle y=\sqrt{(x-a)(b-x)}$ . Taking squares we see that $\displaystyle y$ represents a circle centered $\displaystyle ((a+b)/2,0)$ and radius $\displaystyle r=\sqrt{(a-b)^2/4}$ . So the integral $\displaystyle I$ is half of the area of the corresponding disk, that is $\displaystyle I=(1/2)\pi (a-b)^2/4}=\pi (a-b)^2/8$ .
P.S. Prove It: surely you have a computation mistake, the integral must be positive.
It shouldn't make any difference if you substitute $\displaystyle \displaystyle \left(\frac{a-b}{2}\right)\cos{\theta}$ because $\displaystyle \displaystyle \left(\frac{a-b}{2}\right)\cos{\theta} = -\left(\frac{b-a}{2}\right)\cos{\theta}$, and when you make the substitution, it gets squared anyway, giving the same value.
You are correct that $\displaystyle \displaystyle \frac{1}{2}\cos{2\theta}$ should integrate to $\displaystyle \displaystyle \frac{1}{4}\sin{2\theta}$, but that sign error doesn't make any difference since it goes to 0 from substituting both terminals anyway.
I have found what my mistake is. Since $\displaystyle \displaystyle b \geq a$, that means when I took the square root, I should have done $\displaystyle \displaystyle \sqrt{\left(\frac{a-b}{2}\right)^2} = -\left(\frac{a-b}{2}\right)$, not $\displaystyle \displaystyle \frac{a-b}{2}$ to get the positive value.