I just took a look at your book, and it seemed we can just substitute elements of $\displaystyle F$ whenever we want. In this case, our job is quite easy actually. To see that

$\displaystyle \mathcal{S}=\{p(x)\in F[x]|p(a)=0 \quad \forall a\in F\}$

is an ideal, we want to show that it is closed under addition by elements in $\displaystyle \mathcal{S}$ and closed under multiplication by elements of $\displaystyle F[x]$. For the first bit, suppose $\displaystyle p(x),q(x)\in \mathcal{S}$. Then $\displaystyle (p+q)(a)=p(a)+q(a)=0+0=0$ for all $\displaystyle a\in F$. Thus $\displaystyle (p+q)(x)\in \mathcal{S}$. For the second assertion, suppose $\displaystyle p(x)\in \mathcal{S},r(x)\in F[x]$. Then $\displaystyle r(a)p(a)=r(a)\cdot 0 = 0$ for all $\displaystyle a\in F$. Therefore $\displaystyle r(x)p(x)\in \mathcal{S}$.