That is, of course, the same as
$\displaystyle \begin{bmatrix}5- 1 & 13 \\ 3 & 18- 1\end{bmatrix}\begin{bmatrix}a \\ b\end{bmatrix}$$\displaystyle = \begin{bmatrix}4 & 13 \\ 3 & 17\end{bmatrix}\begin{bmatrix}a \\ b\end{bmatrix}= \begin{bmatrix}0 \\ 0\end{bmatrix}$.
But now the determinant is 3 modulo 26 and 3 is not a "0 divisor" modulo 26 the only solution is the trivial solution, a= b= 0.
More specifically, we can write that as the two equations 4a+ 13b= 0 and 3a+ 17b= 0. If we multiply the first equation by 3 we have 12a+ 13b= 0. If we multiply the second equation by -4 we have -12a- 22b= 0. Adding the two equations, 17b= 0 and, again, since 17 is not a "0 divisor" modulo 26, the only solution is a= b= 0.