Obviously, (0,0) makes both ln(1+ 2x+ 2y) and x/2- y- x^2/2 equal to 0 so (0,0) is a critical point.
To show that it is unstable, you can linearize the problem about (0,0).
which is equal to 2 at (0,0). Similarly, which is 2 at (0,0). ln(1+ 2x+ 2y) can be approximated around (0,0) by its tangent plane there, z= 2x+ 2y.
which is at (0,0). . can be approximated around (0,0) by its tangent plane there, z= x/2- y.
The linearization of this system about (0,0) is
dx/dt= 2x+ 2y and dy/dt= x/2- y.
Now there are a number of ways to show that (0,0) is an unstable critical point for this linearized system and so for the original system.
The "hard way" would be to just go ahead and solve the system! Differentiating the first equation with respect to t gives . Setting dy/dt in that equal to x/2- y gives . But the first equation says that 2y= dx/dt- 2x so that is so the equation is . That is a linear equation with constant coefficients which has characteristic equation . If either of the roots of that is positive, the solution will involve for positive which is unbounded for large t and so (0, 0) is unstable.
The easier way would be to write the system as a matrix equation:
and find the eigenvalues of the coefficient matrix, which will be the same as the solutions to the previous characteristic equation. If either is positive, (0, 0) is an unstable critical point.