Find the condition that

$ax^2 + 2hxy + by^2$

is resolvable into linear factors of the form $(y - mx)$ and $(my + x)$.

Not a clue on this one....

2. \displaystyle \begin{align*}(y - mx)(my + x) &= my^2 + xy - m^2xy - mx^2 \\ &= -mx^2 + (1 - m^2)xy + my^2\end{align*}

If this is equal to $\displaystyle ax^2 + 2hxy + by^2$ then $\displaystyle a = -m, 2h = 1 - m^2, b = m$.

3. But the question says that there must be be only ONE single condition , not three . But from the three given by you...it can be converted into two...which are.."Since $b = m$ and $a = - m$ , $a + b = 0$." and $2h = 1 - m^2$ . The answer given at the back of the book is $a + b = 0$ . But the $2h = 1 - m^2$ can't simply be ignored , can it ?

4. I've got it !

If $(y - mx)$ be a factor of $ax^2 + 2hxy + by^2$ then it must vanish for $y = mx$.

Putting $y = mx$ in $ax^2 + 2hxy + by^2$ , we get $ax^2 + 2hmx^2 + b(m^2)(x^2) = 0$ , or $a + 2hm + bm^2 = 0$ .

Similarly if $( my + x)$ be a factor of $ax^2 + 2hxy + by^2$ then it must vanish for $x = - my$.

Putting $x = - my$ in $ax^2 + 2hxy + by^2$ , we get $a(m^2)(y^2) - 2hmy^2 + by^2 = 0$ or $am^2 - 2hm + b = 0$ .

Then since $m$ satisfies both the equations

$bm^2 + 2hm + a = 0$..................... $(1)$
and, $am^2 - 2hm + b = 0$....................... $(2)$

The condition which is required for both the above equations to have both the roots common is actually the required condition .

On taking some determinants we easily get that, the required condition is

$\frac{a}{b } = \frac{-2h}{2h} = \frac{b}{a}$
$\Rightarrow \frac{a}{b } = -1 = \frac{b}{a}$
$\Rightarrow a + b = 0$ Q.E.D