A nine-point circle goes through 9 specific points relating to a triangle. Three of these points are the midpoints of the triangle's sides. From the picture you posted of the 'green' triangle, it appears the vertices of the triangle are , , and .
The midpoints of these three sides are , , and .
Find the center of the circle by using these three points on the circle. First, find the midpoints for two of the segments connecting these points and get look at the lines perpendicular to these segments. The intersection of these lines is the center of the circle. You can then use the Pythagorean theorem (distance formula) to find the distance this point is from the origin.
I haven't fully analyzed this process, but I would think the formula will either give undefined or invalid results in certain circumstances when the triangle fails to exist (like when ). You might also be able to use this process to show that the center of the circle is always on the altitude of the triangle passing through the origin.