Hi,

a.)

(-2a+3b).(5a-b)

=(-2)(5)+(3)(-1)

=-10-3

=-13

b.)

Give point A(3,1,0); B(2,-3,1): C(-1,0,4); D(5,-6,-2)

A=3i+j; B=2i-3j+k; C=-i+4k; D=5i-6j-2k

5i-6j-2k=x(3i+j)+y(2i-3j+k)+z(-i+4k)

5i-6j-2k=(3x+2y-z)i+(x-3y)j+(y+4z)k

here equation the like vectors

3x+2y-z=5

x-3y=-6

y+4z=-2

solve x , y and z values

and substitute in one of the equation

if its satisfies then the given vectors are coplanar

all the best