Considering a constrained nonlinear programming (NLP) problem
Where and is twice continuously differentiable. The feasible region . It is known that if is convex and is affinely linear for , is a convex set. However, in my problem, and is indefinite for . So I would like to ask if there is any theory may answer the following two questions:
(1)For any twice continuously differentiable but indefinite function , on what condition, is convex in a neighborhood of a point ? (A guess is that Hessian of at is positive semidefinite. Is that the case?)
Just like the image above. The function is indefinite for all , but is locally convex in the neighborhood of , which is .
(2)On what condition, a neighborhood in of a feasible point is a convex set? (I suppose a sufficient condition is that every and is convex in a neighborhood of . But is that necessary?)
Just like the image above. The set is not convex, but is locally convex in the neighborhood of (the red triangle set).