Actually I worked it,
you just the formula given for B
3x1y1-x1y2-x2y1+5x2y2
BY DEFINITION:
$B((x_1,x_2),(y_1,y_2)) = 3x_1y_1 - x_1y_2 - x_2y_1 + 5x_2y_2$
and $\|(x_1,x_2)\|_B \stackrel{\text{def}}{=} \sqrt{B((x_1,x_2,x_1,x_2))}$
$= \sqrt{3(x_1)^2 - (x_1x_2) - (x_2x_1) + 5(x_2)^2} = \sqrt{3(x_1)^2 - 2(x_1x_2) + 5(x_2)^2}$.
So, if $x_1 = 4$ and $x_2 = 3$, we have:
$\|(4,3)\|_B = \sqrt{3(16) - 2(12) + 5(9)} = \sqrt{48 - 24 + 45} = \sqrt{69}$.
Also by definition, $(x_1,x_2),(y_1,y_2)$ are orthogonal with respect to $B$, if:
$B((x_1,x_2),(y_1,y_2)) = 0$. All we need to do is compute this:
$B(1,1),(2,-1)) = 3(1\ast2) - (1\ast(-1)) - (1\ast2) + 5(1\ast(-1)) = 3(2) - (-1) - 2 + 5(-1) = 6 + 1 - 2 - 5 = 0$
*****************
If you prefer, we may define $B((x_1,x_2),(y_1,y_2))$ as:
$\begin{bmatrix}x_1&x_2 \end{bmatrix} \begin{bmatrix} 3&-1\\-1&5 \end{bmatrix} \begin{bmatrix}y_1\\y_2 \end{bmatrix}$,
the computations remain the same.