I assume that R means , the set of real numbers.

(If R stands for a generic ring, you'll need to specify that it's a domain.)

I also assume that your "b = -11" is a typo - that you intended "b = -1"

a*b = -1 implies ab + a + b = -1.

Now write that as "a, b stuff" = 0, and it will actually factor, and from there lead you to your result.