from the first equation we have:
a = 2b + 6c + 12 (mod 13) (*)
therefore:
8(2b + 6c + 12) + 3b + 3c = 9 (mod 13)
3b + 9c + 5 + 3b + 3c = 9 (mod 13)
6b + 12c = 4 (mod 13)
6b = c + 4 (mod 13)
11(6b) = 11(c + 4) (mod 13)
b = 11c + 5 (mod 13)
substituting this back in (*), we get:
a = 2(11c + 5) + 6c + 12 = 9c + 10 + 6c + 12 = 2c + 9 (mod 13)
this agrees with earboth's answer, if you reduce mod 13:
a = 3(21 - 4c)/85 = 3(8 - 4c)/7 = 6(8 - 4c) = 6(8 + 9c) = 2c + 9
b = (87 - 53c)/85 = (9 - c)/7 = 2(9 - c) = 2(9 + 12c) = 11c + 5
finally, we turn to the last equation:
a^{2} + a + bc = 12
(2c + 9)^{2} + 2c + 9 + (11c + 5)c = 12 (mod 13)
4c^{2} + 10c + 3 + 2c + 9 + 11c^{2} + 5c = 12 (mod 13)
2c^{2} + 4c + 12 = 12 (mod 13)
2c^{2} + 4c = 0 (mod 13)
c(2c + 4) = 0 (mod 13)
whereby we see that the two solutions are c = 0, and c = 11. since a quadratic may introduce extraneous solutions, we must check each one separately in the original equations.
c = 0 leads to: a = 9, and b = 5.
9 + 11(5) = 9 + 3 = 12 (mod 13), so this is consistent with the first equation.
8(9) + 3(5) = 7 + 2 = 9 (mod 13), so this is consistent with the second equation.
(9)^{2} + 9 = 3 + 9 = 12 (mod 13), so this is consistent with the third equation. thus (a,b,c) = (9,5,0) is a solution.
c = 11 leads to: a = 5, and b = 9.
5 + 11(9) + 7(11) = 5 + 8 + 12 = 12 (mod 13), so this checks out.
8(5) + 3(9) + 3(11) = 1 + 1 + 7 = 9 (mod 13), so this checks out as well.
(5)^{2} + 5 + (9)(11) = 12 + 5 + 8 = 12 (mod 13), so this also checks out. thus (a,b,c) = (5,9,10) is also a solution.
note: check my arithmetic, as these computations have been somewhat involved, and i may have made errors.
Never mind...ok now; thanks to:
Finite field arithmetic - Wikipedia, the free encyclopedia