The point must satisfy three inequalities; it must lie on the proper sides of all three lines making up the triangle.
For example, consider the line connecting (-1, 7) and (8, -4), which is (-4 - 7)/(8 + 1) = (y - 7)/(x + 1) or y = -(11/9)x + 5 + 7/9. To find the proper side for this line, test the third point (0, 0) to get the inequality y < -(11/9)x + 5 + 7/9.
Do the same for the remaining two lines.
A point is on the boundary if it satisfies the line equation, and is outside the triangle if it does not satisfy one of the inequalities and does not satisfy any equality.