To be honest, I cannot say that I really follow what you are trying to do.
But if the polygon is convex the you may explore lines of support.
A convex polygon has all sides on the same side of a line of support.
I suggest that you do a search on the terms “convex” and “line of support”.
For a non-convex polygon, I think one would have to reduce it to convex sub-polygons.