I went to my prof for help on this and he couldn't figure it out.

I am trying to determine the values of k for which V(x,y)=x^2+ky^2 is Lyapunov for the following first order system:

x'=-x+y-x^2-y^2+(x)(y^2) ; y'=-y+xy-y^2-(x^2)(y)

I'd also like to like to be able to say something about the domain of stability for the case k=1. So far I've intuitively determined that there is a neighborhood around (0,0) for which the Lyanpunov's(k=1) derivative is always positive, thus unstable, thus there is no domain of stability for the case k=1. I'm having some trouble showing it in a rigorous manner, though.

Any help would be greatly appreciated.