If two vectors are perpendicular to each other, then their dot product is equal to 0.
So, find the vector AB, then find the dot product of the vectors AB and OC. If this gives zero, then it means that AB and OC vectors are perpendicular to each other.
For the second part, since you just showed that vector OC is perpencicular to vector AB, you already have showed that the shortest distance from O to AB 'lies' on the line OC. So, find the point where line OC and AB meet, and from there, you can find the distance from their point of intersection to the origin, which is also the shortest distance from O to line AB.
I hope I didn't confuse you too much