A conic which gived by x^2+xy+y^2=3 Rotated 45 degrees about the origin counterclockwise. What is the quation is transformed to?
If we rotate the $x$ and $y$ axes (along with our conic) to new axes $x'$ and $y'$, then the old coordinates, in terms of the new ones, are:
$x = x'\cos 45^{\circ} + y'\sin 45^{\circ} = \dfrac{\sqrt{2}}{2}(x' + y')$
$y = y'\cos 45^{\circ} - x'\sin 45^{\circ} = \dfrac{\sqrt{2}}{2}(y' - x')$
So the equation in our new coordinates is:
$\left( \dfrac{\sqrt{2}}{2}(x' + y')\right)^2 + \left(\dfrac{\sqrt{2}}{2}(x' + y')\right)\left(\dfrac{\sqrt{2}}{2}(y' - x')\right) + \left(\dfrac{\sqrt{2}}{2}(y' - x')\right)^2 = 3$ or:
$\dfrac{1}{2}(x'^2 + 2x'y' + y'^2) + \dfrac{1}{2}(y'^2 - x'^2) + \dfrac{1}{2}(y'^2 - 2x'y' + x'^2) = 3$, and cancelling and collecting terms:
$x'^2 + 3y'^2 = 6$
Compare these two plots:
x^2+xy+y^2=3 - Wolfram|Alpha
x^2 + 3y^2 = 6 - Wolfram|Alpha
Your book is right, I mis-read the question and rotated it CLOCKWISE.
If we rotate it counter-clockwise, the formula for $x',y'$ are:
$x = x'\cos 45^{\circ} - y'\sin 45^{\circ} = \dfrac{\sqrt{2}}{2}(x' - y')$
$y = y'\cos 45^{\circ} + x'\sin 45^{\circ} = \dfrac{\sqrt{2}}{2}(x' + y')$
which leads to:
$\left( \dfrac{\sqrt{2}}{2}(x' - y')\right)^2 + \left(\dfrac{\sqrt{2}}{2}(x' = y')\right)\left(\dfrac{\sqrt{2}}{2}(x' + y')\right) + \left(\dfrac{\sqrt{2}}{2}(x' + y')\right)^2 = 3$
$\dfrac{1}{2}(x'^2 - 2x'y' + y'^2) + \dfrac{1}{2}(x'^2 - y'^2) + \dfrac{1}{2}(x'^2 + 2x'y' + y'^2) = 3$
$3x'^2 + y'^2 = 6$, as your book says, with this plot:
3x^2 + y^2 = 6 - Wolfram|Alpha
I apologize for the mix-up, I "turned it the wrong way"