To solve this we first make the observation that if $\displaystyle A\in\text{Mat}_n\left(\mathbb{Z}\right)$ then $\displaystyle A$ is invertible and $\displaystyle A^{-1}\in\text{Mat}_n\left(\mathbb{Z}\right)$ if and only if $\displaystyle \det A=\pm 1$. Indeed, clearly if $\displaystyle \det A=\pm 1$ the result follows. Conversely, the conclusion follows by noticing that since $\displaystyle A^{-1}\in\text{Mat}_n\left(\mathbb{Z}\right)$ that $\displaystyle \det A^{-1}=\frac{1}{\det A}\in\mathbb{Z}$ from where the conclusion follows.

So, with this observation let $\displaystyle p(x)=\det (A+xB)$. Then, since $\displaystyle A,B\in\text{Mat}_2\left(\mathbb{Z}\right)$ we see that $\displaystyle \deg p\leqslant 2$. Note though that by assumption of the problem and the above observation we have that $\displaystyle p(x)\in\{-1,1\},\quad x\in\{0,1,2,3,4\}$. But, by the Pigeon Hole principle we know that $\displaystyle p(x)$ takes the value either $\displaystyle 1$ or $\displaystyle -1$ at least three times on $\displaystyle \{0,1,2,3,4\}$. But, since $\displaystyle \deg p\leqslant 2$ this implies that $\displaystyle p$ is constant and $\displaystyle p(x)=1$ or $\displaystyle p(x)=-1$. Thus, by our observation we may conclude that $\displaystyle A+xB$ is invertible and $\displaystyle \left(A+xB\right)^{-1}\in\text{Mat}_2\left(\mathbb{Z}\right)$ for every $\displaystyle x\in\mathbb{Z}$. $\displaystyle \blacksquare$